KR20190115930A - Method, apparatus and computer program for searching image - Google Patents

Abstract

The present invention relates to a method, a device and a computer program for searching an image, wherein the image searching method searches a similar image by combining one or more search images based on individual search criterion properties by using one or more search images inputted by a user and each of the search criterion properties thereof when searching for a similar image to the search image inputted by the user. To be more specifically, the disclosed image searching method comprises: a search information receiving step of allowing a server to receive a plurality of search data information including one or more search images inputted by a user from a terminal; a similar image searching step of searching for a similar image by combining search data part information corresponding to each of the search property information for the plurality of search data information from the plurality of search data information; and a similar image providing step of transmitting similar image information on the searched similar image to the terminal. According to the present invention, a user can search a desired image conveniently and accurately.

Description

Method, apparatus and computer program for searching image

The present invention relates to a method, an apparatus, and a computer program for searching an image, and more particularly, by using a plurality of search data information including one or more search images input by a user and a plurality of search attribute information thereof. The present invention relates to an image retrieval method, apparatus, and computer program characterized by retrieving a similar image by combining portions corresponding to the respective retrieval attribute information in each retrieval data information.

With the development of information and communication technology, information retrieval using various terminals such as smart phones and personal computers (PCs) has been widely utilized.

In particular, in the past, mainly provided by searching for a document including the text based on the text, such as a search word entered by the user, but in recent years, users are increasingly searching for an image such as a picture or a picture. This is because it is often more efficient to see images such as photographs rather than acquiring information using text alone, as in the words 百 聞 不 如一 見). However, when it is not easy to clearly describe the characteristics by text alone, such as clothing or scenery, the user may have difficulty in finding an image desired by the user. For example, even when a user wants to search for clothing of a style that he / she thinks of, he or she often encounters a difficulty in finding an appropriate image by entering text as a search word.

On the other hand, recently, a service for receiving a search image instead of text from a user and searching for and providing an image similar to the input search image has been attempted. However, in this case, only the image most similar to the search image input by the user may be selected and provided, and similar images may not be searched and provided based on a specific property of the search image. That is, in the past, the most similar image could be searched considering all the shapes, colors, and textures of the objects included in the input search image, but similar images were searched based on a specific property of the input search image. Or, it is difficult to search for an image based on a combination of specific attributes of a plurality of input search images.

For example, in the past, when a user inputs a red horizontal stripe T-shirt image as a search image, the user may search for the image most similar to the input red horizontal stripe T-shirt image, but similar to the T-shirt included in the search image. It was difficult to search for images of T-shirts with different colors or different patterns, and furthermore, by combining the T-shirt form of the first search image and the pattern of the second search image, the second search was performed on the T-shirt form of the first search image. As in the case of searching for an image having a pattern of an image, it was more difficult to search for a similar image by combining properties of objects included in several search images.

Accordingly, in searching for an image, not only an image can be searched and provided based on a specific property of a given search image, but also a method of searching and providing a similar image by combining specific properties of a plurality of searched images. With this, users will be able to search for the image they want more conveniently and accurately, but there is no proper solution for this.

Republic of Korea Patent Publication No. 10-2007-0009338 (published January 18, 2007)

The present invention was devised to solve the above problems of the prior art, and in searching and providing an image similar to the search image input by the user, the similar image is based on a specific property of the search image input by the user. An object of the present invention is to provide an image retrieval method, apparatus, and computer program that can be searched and provided.

Furthermore, an object of the present invention is to provide an image retrieval method, apparatus, and computer program capable of searching for and providing similar images by combining specific attributes of a plurality of search images input by a user.

An image retrieval method according to an aspect of the present invention for solving the above problems, the server receives a search information receiving a plurality of search data information including one or more search images input by the user; A similar image retrieval step of retrieving a similar image by combining retrieval data partial information corresponding to respective retrieval attribute information for the plurality of retrieval data information from the retrieval data information; And providing similar image information for transmitting the similar image information about the found similar image to the terminal.

In addition, the image search method according to another aspect of the present invention, the terminal information transmission step of transmitting a plurality of search data information including at least one search image input by the user to the server; A similar image information receiving step of receiving, from the server, similar image information for a similar image found by combining search data partial information corresponding to respective search attribute information for the plurality of search data information in the plurality of search data information; And providing similar image information for providing the similar image information on the similar image to the user.

A computer program according to another aspect of the present invention is a computer program stored in a computer readable medium for executing each step of the image retrieval method described above on a computer.

According to still another aspect of the present invention, a server includes: a search information receiver configured to receive a plurality of search data information including at least one search image input by a user from a terminal; A similar image search unit for searching for a similar image by combining search data partial information corresponding to respective search attribute information of the plurality of search data information in the plurality of search data information; And similar image providing unit which transmits similar image information about the retrieved similar image to the terminal.

According to another aspect of the present invention, a terminal may include: a search information transmitter configured to transmit a plurality of search data information including one or more search images input by a user to a server; A similar image information receiver configured to receive similar image information of a similar image found by combining search data partial information corresponding to respective search attribute information of the plurality of search data information in the plurality of search data information from the server; And similar image information providing unit which provides similar image information on the similar image to the user.

In the image retrieval method, apparatus, and computer program according to an embodiment of the present invention, in searching for and providing an image similar to a given search image, the user searches for and provides a similar image based on a specific attribute of the given search image. You will be able to search for the images you want more conveniently and accurately.

In addition, in the image retrieval method, apparatus and computer program according to an embodiment of the present invention, by searching for and providing an image similar to the combination of the plurality of search images based on the respective search attributes for a plurality of search images, A similar image for the image combining the plurality of search images can be provided more conveniently and accurately.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, included as part of the detailed description in order to provide a thorough understanding of the present invention, provide examples of the present invention and together with the description, describe the technical idea of the present invention.
1 is a block diagram of an image retrieval system according to an embodiment of the present invention.
2 is a flowchart of an image retrieval method from a server perspective according to an embodiment of the present invention.
3 is an exemplary view of a search information input screen according to an embodiment of the present invention.
4 is a diagram illustrating a similar image search result using a plurality of search images according to an embodiment of the present invention.
5 is an exemplary view of a search information input screen according to another embodiment of the present invention.
6 is a diagram illustrating a similar image search result using a search image and a text input according to an embodiment of the present invention.
7 is an exemplary view of a search information selection screen according to an embodiment of the present invention.
8 and 9 are detailed flowcharts of an image retrieval method according to an embodiment of the present invention.
10 to 12 are diagrams for describing a detailed operation of an image retrieval method according to an embodiment of the present invention.
13 is a diagram illustrating an operation of an image retrieval system according to an embodiment of the present invention.
14 to 15 illustrate a method of calculating an index feature vector according to an embodiment of the present invention.
16 is a flowchart illustrating an image retrieval method from a terminal perspective according to an embodiment of the present invention.
17 is a detailed flowchart of an image retrieval method according to an embodiment of the present invention.
18 is a block diagram of a server for searching for an image according to an embodiment of the present invention.
19 is a block diagram of a terminal for image search according to an embodiment of the present invention.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be described in detail with reference to the accompanying drawings.

In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as 'first' and 'second' may be used to describe various components, but the components are not limited by the terms, and the terms are only used to distinguish one component from another component. Used.

Hereinafter, exemplary embodiments of an image retrieval method, apparatus and computer program according to the present invention will be described in detail with reference to the accompanying drawings.

First, FIG. 1 shows a configuration diagram of an image retrieval system 100 according to an embodiment of the present invention. As shown in FIG. 1, the image search system 100 according to an embodiment of the present invention transmits a plurality of pieces of search data information including one or more search images input by a user to the server 120. The terminal 110 receives the similar image information of the similar image retrieved from the server 120 and provides the same to the user, and a plurality of pieces of search data information including one or more search images input by the user from the terminal 110. In response to the transmission, the similar image is searched by combining search data partial information corresponding to each search attribute information of the plurality of search data information from the plurality of search data information, and the similar image information of the similar image is searched for by the terminal ( It may be configured to include a server 120 for transmitting to 110 and the network 130 for connecting the terminal 110 and the server 120.

In this case, as the terminal 110, various portable terminals such as a smart phone, a tablet PC, a PDA, a mobile phone, etc. may be used. In addition, the terminal 110 may receive search information from a user such as a personal computer (PC), a notebook PC, and receive the search information. Various types of terminals may be employed that may be used to provide an image retrieved using.

In addition, the image in the present invention includes a picture, a picture, and the like, and may further be configured in various ways including a character, a symbol, a figure, and the like.

In addition, the network 130 connecting the terminal 110 and the server 120 may include a wired network and a wireless network. Specifically, a local area network (LAN) and a metropolitan area network (MAN: Metropolitan) And various communication networks such as an area network (WAN) and a wide area network (WAN). In addition, the network 130 may include a known World Wide Web (WWW). However, the communication network 130 according to the present invention is not limited to the networks listed above, and may include at least part of a known wireless data network, a known telephone network, or a known wired / wireless television network.

2 shows a flowchart of an image retrieval method from a server 120 perspective according to an embodiment of the present invention. As can be seen in Figure 2, the image search method according to an embodiment of the present invention, the server 120 receives a plurality of search data information including one or more search images input by the user from the terminal 110 Search image receiving step (S110) to perform a similar image search step (S120) to search for a similar image by combining the search data portion information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information (S120) And providing similar image information of the searched similar image to the terminal (S130).

Hereinafter, the image retrieval method according to an embodiment of the present invention will be described in detail with reference to FIG. 2.

First, in the search information receiving step (S110), the server 120 receives a plurality of search data information including one or more search images input by the user from the terminal 110. In this case, the server 120 may receive a plurality of search attribute information for the plurality of search data information together with the plurality of search data information from the terminal 110.

As a specific example, as shown in FIG. 3, a user inputs a plurality of search images 101, 102, and 103 to the terminal 110 to search for the plurality of search images 101, 102, and 103. Data information may be transmitted to the server 120.

In this case, the plurality of search images 101, 102, 103 itself may be transmitted as the search data information, but URL information of the plurality of search images 101, 102, 103 may be transmitted to the server 120. The plurality of search image files 101, 102, 103 may be collected from a device other than the terminal 110 using the URL information.

In addition, the user may directly input the searched images 101, 102, and 103 by using his or her own terminal, or select an image generated by himself or another user.

In addition, the user may input search text in addition to the search images 101, 102, and 103 as the search data information.

In addition, search attribute information (e.g., "shape", "pattern", "color" in FIG. 3) for a plurality of pieces of search data information such as the search image is set in advance so that the user searches for the corresponding search attribute information. An image or the like may be input, or a user may input a search image or the like and set a search attribute for the input search image or the like. For example, in FIG. 3, the user sets "shape" as search attribute information for the first search image 101 (104 in FIG. 3), and "pattern" as search attribute information for the second search image 102. 3 is set (105 in FIG. 3), and as the search attribute information for the third search image 103, " color " is set (106 in FIG. 3).

Subsequently, in the similar image retrieval step (S120), the server 120 searches for similar images by combining search data partial information corresponding to the respective search criterion attribute information in the respective search data information.

That is, referring to the case of FIG. 3, since the search attribute information of the first search image 101 input by the user is “shape”, the first search image 101 excludes “pattern” or “color”. Information about the “shape” of the shirt included in the first search image 101, similarly, “pattern” information is taken from the second search image 102, and “ Similar to an image that takes and combines the "color" information to combine the "shape" of the first search image 101 with the "pattern" of the second search image 102 and the "color" of the third search image 103. The image will be searched.

Accordingly, as shown in FIG. 4, when a plurality of search images ((a), (b), and (c) of FIG. 4) are input, a plurality of search criteria attribute information for the plurality of search images ( For example, an image combining the plurality of search images based on "shape", "pattern", "color" ("shape" of the first search image 101 and "" of the second search image 102). A similar image for the " pattern " and the " color " of the third search image 103.

Accordingly, as shown in FIG. 4D, the "shape" of the first search image 101, the "pattern" of the second search image 102, and the "color" of the third search image 103. It can be seen that similar images including the characteristic of "can be searched for.

Subsequently, in the providing of the similar image (S130), the server 120 transmits similar image information about the found similar image to the terminal 110.

In this case, the server 120 may transmit the searched similar image itself as the similar image information, but the URL information of the searched similar image is transmitted, and thus the terminal 110 uses the URL information to transmit the similar image. It may be to collect from a device other than the server 120.

In addition, the present invention will be mainly described below using examples such as "clothing", but the present invention is not limited thereto. In addition, the present invention may combine specific attributes of a plurality of pieces of search data information such as a search image or a search keyword. It can be used in various ways as an application for searching for similar images.

In addition, the image search method according to an embodiment of the present invention, not only can search for similar images using a plurality of search images input by the user, but also search for one or more search images and one or more search texts input by the user. You can also search for similar images.

For example, FIG. 5 illustrates a case in which the user inputs the second search text 301 together with the first search image 302.

At this time, in the search information receiving step (S110), the server 120, the first search image 302 input by the user from the terminal 110 and the first search attribute information 305 and the first search image, 2, search information including the search text 301 and the second search criteria information 304 is received. Accordingly, in the similar image search step (S120), the server 120 receives the first search image. First search data portion information corresponding to the first search attribute information ("pattern" in 305 of FIG. 5) at 302, i.e., "pattern" of the first search image 302 of FIG. The second search data portion information (i.e., "shape" of the skirt) corresponding to the second search attribute information ("shape" in 304 of FIG. 5) is combined with respect to the second search text 301, and FIG. The similar image is searched for an image that combines the "pattern" of the first search image 302 and the "shape" of the skirt.

In addition, since the input of the search image or the search text for "color" (306 in FIG. 5) is omitted (303 "pass" in FIG. 5), as shown in FIG. 5, the "color" in searching similar images. Similar images can be searched based only on "shape" and "pattern" without any restriction on.

Accordingly, as shown in FIG. 5, when one first search image (FIG. 5A) and one second search text (“skirt”) are input, the first search image and the second search image are input. Based on the respective search criteria attribute information for the search text (in FIG. 4, "pattern" 305 for the first search image 302, "shape" 304 for the second search text 301). Search for similar images for the image combining the first search image and the partial information of the second search text (the image having the "pattern" of the first search image 302 and the "shape" of the second search text 301) Done.

In addition, when a search image is input in the present invention, after detecting a predetermined object among the search images, an image search may be performed using the detected object. As can be seen in Figure 6 (a), if there is a region that does not correspond to the object of the input search image, this may cause a obstacle in obtaining a desired search image, the object area of the input search image After detection (401 of FIG. 6), when the image search is performed using the detected object area (Fig. 6 (b)) it is possible to derive a more accurate search results.

Accordingly, as shown in FIG. 6C, the "shape" of the first search image 101, the "pattern" of the second search image 102, and the "color" of the third search image 103. It can be seen that similar images including the characteristic of "can be searched for.

In addition, in the present invention, when only the search image and search text for "shape" and "pattern" are input, and the search image or search keyword for "color" is not input, it can be seen in FIG. As described above, it can be seen that a skirt having a similar pattern and a completely different color can be searched according to the input search image and the search text.

In addition, in the image retrieval method according to an embodiment of the present invention, the terminal 110 may provide a user interface for selecting a search image stored in advance by the user and search attribute information thereof. The user may search for similar images more conveniently using previously stored search images and search attribute information thereof.

More specifically, as shown in FIG. 7, the terminal 110 may provide the user with a "pattern" list 502 stored in advance so that the user may select 201. To this end, when the user takes a picture of an object, it provides an interface for selecting search attribute information (for example, providing a menu for selecting "shape", "pattern", "color", etc.). In addition, the search attribute information and the photo selected by the user may be stored in the terminal 110 or transmitted to and stored in the server 120, and then may be easily selected by the user later. In this case, the picture may be stored as the image itself, but may be converted into a feature and the like to represent the feature.

Furthermore, a function of recommending using the user's selection history, taste, profile, statistics, etc. may be added while providing the user with the search image and the search attribute information thereof.

For example, if a user usually searches a lot of polka dots dresses, even if the user does not specify a search image or search text for the pattern item, the user may recommend the search results by adding a polka dot condition. Alternatively, the search results may be recommended to the user using an item corresponding to a trendy item that the user searched a lot in the last month and a search condition therefor. In this case, the items searched by many users may be selected based on behavior patterns of all other users who use the service, and may be selected in consideration of behavior patterns of users similar to the age, gender, and taste of the target user. have.

In addition, the user may designate a list of target articles of an image to be searched to perform an image search within the designated category range. For example, the user may designate a list of items to be searched through text input or list selection, such as "clothing" or "furniture". In this case, the present invention provides the search image within the range of the specified list of items to be searched. Or an image is searched using a plurality of search data information such as search text and a plurality of search attribute information thereof.

Furthermore, in the image retrieval method according to an embodiment of the present invention, after calculating a plurality of search image attribute feature vectors corresponding to the plurality of search attribute information from the input plurality of search images, the calculated search image A similar image may be searched for in a plurality of stored images stored in the database 124 using the feature-specific feature vector.

In this case, the database 124 stores the feature vectors for each of the stored image attributes, which are calculated and stored with respect to the plurality of stored attribute information, for the plurality of stored images together with the plurality of stored images. Accordingly, in the image retrieval method according to an embodiment of the present invention, the feature vectors for each of the plurality of search image attributes calculated from the plurality of search images are compared with the feature vectors for each of the stored image attributes stored in the database 124. The image similar to the image obtained by combining the plurality of search images may be searched and provided based on the search attribute information of the plurality of search images.

More specifically, as shown in Figure 8, in the image search method according to an embodiment of the present invention, the similar image search step (S120) corresponds to the plurality of search attribute information in the plurality of search images A plurality of search image property-specific feature vector calculating steps (S121) and a plurality of stored images stored in the database 124 are calculated based on the plurality of search property information and are stored. The method may include a database search step (S122) of searching similar images by comparing the feature vector for each stored image attribute with the feature vector for each of the plurality of search image attributes.

Further, as shown in FIG. 9, the step of calculating a feature vector for each search image property (S121) may include detecting one or more objects included in the search image (S1211) and for each of the one or more objects. A feature vector for each search image property may be calculated based on the search property information (S1212).

Hereinafter, in the image retrieval method according to an embodiment of the present invention with reference to FIGS. 10 to 12, a feature vector for each retrieval image property is calculated from a plurality of retrieval images, and among the plurality of stored images stored in the database 124 using the feature vector. The method of searching similar images will be described in more detail.

First, when a user inputs a method of photographing or uploading a search image 701 (FIG. 10A), the search image 702 is trained using a machine learning technique such as deep learning. It enters into the neural network model and goes through the processing process.

In this case, the neural network model is configured based on a region with CNN (Region with CNN features) algorithm, which is a kind of deep learning technology, and thus detects the position of an object such as a fashion item in the input search image 701. An object detector model may be implemented (702, 703, 704 in FIG. 10B), and the object may be classified according to a predetermined type. More specifically, an YOLO v2 model or the like may be used as the object detector model, and the YOLO v2 model may be implemented by learning using learning data such as a fashion item image having a bounding box. have.

Furthermore, as an embodiment of the present invention, the neural network model may detect a location of an object such as a fashion item in an input image, and then classify the detected object such as a fashion item into a top, a bottom, a bag, shoes, and an outer. .

In addition, in the image retrieval method according to an embodiment of the present invention, an object, such as a fashion item, a pattern, a color, and the like, for an object such as a fashion item included in the search image 701 by using a neural network model It may be implemented as a feature vector extractor model for calculating a feature vector for each search image property according to the specified search attribute information. As shown in FIG. 10C, the detected object ( It is possible to calculate the feature vector 708 for each search image property according to predetermined search property information such as category, texture, and color with respect to 705, 706, and 707.

More specifically, an Inception v3 model or the like may be used as the feature vector extractor model, in which the Inception v3 model uses a pre-trained model for category, texture, and color. For example, fine tuning may be performed using learning data such as a fashion item image based on predetermined search attribute information.

Further, the learning data may be produced by tagging the location and search attribute of the object in a fashion item image collected through web crawling or the like.

For example, when the first search image 801 of FIG. 11 is input, the objects 802, 803, and 804 included in the first search image may be sensed. The feature vector for each search image property is calculated according to the determined search property information. In the first search image of FIG. 11, objects of the upper 802, the lower 803, and the bag 804 have been detected, and then feature vector by attribute of the search image for the upper 802 according to the “shape” search attribute information. (802a) was calculated ({0.22, 0.8280, 0.3311, ...}), and the feature vector 803a for each search image attribute for the bottom 803 was calculated ({0.5213, 0.0027, 0.68, ...). }), It can also be seen that the feature vector 804a for each of the search image attributes of the bag 804 has been calculated ({0.48, 0.0187, 0.241, ...}).

Also, from the other search images input by the user, the feature vector for each search image attribute according to the search attribute information such as "pattern" and "color" is calculated. For example, for the second search image, a feature vector for each search image attribute according to the “pattern” search attribute information is calculated (for example, {0.761, 0.8616, 0.135, ...}), and the third search image is calculated. For example, after calculating a feature vector for each search image property according to the "color" search attribute information (for example, {0.101, 0.1672, 0.583, ...}), the "shape" search for the image of the first search image is performed. Feature vector by search image property ({0.22, 0.8280, 0.3311, ...}) according to attribute information and feature vector by search image property ({0.761, 0.8616) according to "pattern" search attribute information for the second search image , 0.135, ...}) and feature vectors ({0.101, 0.1672, 0.583, ...}) according to the search image attributes according to the "color" search attribute information for the third search image ([{0.22, 0.8280, 0.3311, ...}, {0.761, 0.8616, 0.135, ...}, {0.101, 0.1672, 0.583, ...}], using the "shape" of the first search image It is possible to effectively search for similar images similar to the image in which the "pattern" of the second search image and the "color" of the third search image are combined.

In addition, for the stored image stored in the database 124, as shown in FIG. 12, after detecting the objects included in the stored image, the detected object (901 of FIG. 12) is previously described. The feature vector for each stored image attribute is calculated according to the determined search attribute information.

Accordingly, for the object 901 detected in the stored image of FIG. 12, the characteristic vector 902 for each stored image attribute is calculated for each of the predetermined search attribute information "shape", "pattern", and "color" (shape: {0.22, 0.8280, 0.3311, ...}, Pattern: {0.342, 0.081, 0.0017, ...}, Color: {0.84, 0.5402, 0.7813, ...}).

Accordingly, the feature vector for each stored image attribute of the search attribute information may be stored in the database 124 (903 of FIG. 12) and used to search for a similar image for an image combining a plurality of search images.

In addition, the database 124 may store a connection storage image feature vector to which the plurality of storage image property vectors are connected to the storage image, and in this case, the server 120 may store the feature by the plurality of search image properties. After generating the connection search image feature vector to which the vector is connected, the similar image may be searched based on the similarity with the connection storage image feature vector stored in the database 124.

Accordingly, by combining the feature vectors for each of the search image attributes of the plurality of search images, the stored images having the feature vectors for each of the stored image attributes similar to the combined feature vector for the search image attributes are searched for the plurality of search images. It is possible to effectively search for and provide an image similar to the image combining the plurality of search images based on the respective search attributes.

In addition, FIG. 13 is a view for explaining in more detail the operation of the image search method, apparatus and system 100 according to an embodiment of the present invention.

Hereinafter, operations of the image retrieval method, apparatus, and system 100 according to an exemplary embodiment of the present invention will be described in detail with reference to FIG. 13.

First, as shown in FIG. 13, a user may input search data information such as a plurality of search images 601, 602, and 603 using the terminal 110.

As a more specific example, the user may input the first search image 101, the second search image 102, and the third search image 103 as shown in FIG. 3. In this case, some or all of the plurality of search images 601, 602, and 603 may be subjected to a process of detecting an object (for example, a fashion item) as described in FIG. 6B. (604).

Subsequently, a neural network model or the like is applied to the plurality of search images 601, 602, 603 or the detected object 604 to predetermined search attribute information such as category, texture, and color. The feature vectors for respective search image attributes are calculated (605, 606, 607).

Accordingly, similar images may be searched by combining the calculated feature vectors 605, 606, and 607 of each search image property, but further, an image search method, apparatus, and system according to an embodiment of the present invention. In operation 100, a vector quantization and indexing technique may be applied to search for similar images more quickly from a large storage image.

Accordingly, in the image retrieval method, apparatus, and system 100 according to an embodiment of the present invention, after sorting a plurality of elements included in the feature vector for each search image property by size, the top k having the largest size is obtained. After calculating an index feature vector (608, 609, 610 of FIG. 13) as an element having an index value indicating a position in the feature vector for each search image property of the elements, the index feature vector is used to search the search image. You can also search for similar images.

For example, FIGS. 14 and 15 illustrate a method of searching for similar images by calculating an index feature vector according to an embodiment of the present invention.

First, as shown in FIG. 14, when two images (both search images or stored images) have a similar shape (for example, clothing A and clothing A 'of FIG. 14), the " Shape "criteria (search or store) The feature vectors by image attributes are placed at similar positions (ie, similar index values). On the other hand, when the images have different shapes (for example, clothing B of FIG. 14), the feature vectors for each of the "shape" reference image attributes of the respective images also have significantly different values.

That is, as shown in the example of FIG. 14, when the images have similar shapes (clothing A and clothing A 'of FIG. 14), the feature vectors for each “shape” reference image attribute of the image are also similar to each other. If the element having a large value is located (1001, 1002, 1003 in FIG. 14) and the image has a different shape (for example, clothing B in FIG. 14), the "shape" reference image attribute of the image Star feature vectors also have very different values. Accordingly, when the feature vector for each "shape" reference image attribute of each image shows a similar pattern, the likelihood of each image having the same or similar shape increases.

Accordingly, in the present invention, an index feature vector is formed by calculating a predetermined number of some elements based on the size of the elements among a plurality of elements included in the feature vector for each image property, and includes only the some elements. By calculating the similar image using the index feature vector, it is possible to search similar images quickly and efficiently while reducing computational resources.

Furthermore, as shown in FIG. 15, the index feature vector may be arranged in order of the size of some elements calculated by a predetermined number based on the size in the feature vector for each image property including the plurality of elements. (top-k), the index feature vector may be configured with each index value representing a position of the calculated partial elements of the feature-specific feature vector.

For example, given a feature vector 1101 for each image property for an image in FIG. 15, a predetermined number (eight in FIG. 15) of elements of the feature vector for each image property is determined based on a size. Calculate ("7, 7, 6, 8, 7, 7, 7, 9" in FIG. 15) and sort in that order ("9, 8, 7, 7, 7, 7, 7, in the above example). 6 "), the index value (position) of" 9 "is" 19 ", and the index of" 8 "is" 15 "," 7 "indicating the position of the calculated element-specific feature vectors of the elements. The index of "is" 2, 3, 16, 17, 18 "(sequenced because it is five, and the index of" 6 "is" 11 "), and constitutes an index feature vector (1102 in FIG. 15) for the image ( {19, 15, 2, 3, 16, 17, 18, 11}).

Accordingly, the index feature vector for each image can efficiently judge similarity of each image while having only elements that are significantly reduced than the feature vector for each image property, thereby reducing similar computational resources and reducing similar computational resources. And efficient search.

Accordingly, index feature vectors F1, F2, and F3 for the plurality of search images 601, 602, and 603 may be calculated 608, 609, and 610, and may also be stored in a database (611 of FIG. 13). Since the index feature vector for the image is stored, a stored image having an index feature vector similar to the case of combining the index feature vectors for the plurality of search images (601, 602, 603) (F1, F2, F3) is obtained. By searching, it is possible to search for and provide an image similar to an image combining the plurality of search images based on the search attributes of the plurality of search images 601, 602, 603.

In this case, in the case of combining the index feature vectors of the stored images stored in the database 611 with the index feature vectors for the plurality of search images 601, 602, 603 (F1, F2, F3), It is desirable to use index-based search techniques such as Matched Term Count (MTC), TF-IDF, and BM25.

16 illustrates a flowchart of an image retrieval method from the perspective of the terminal 110 according to an embodiment of the present invention. The image retrieval method from the terminal 110 perspective according to an embodiment of the present invention will be inferred from the description of the image retrieval method from the server 120 perspective according to the embodiment of the present invention described above. Where possible, more detailed description will be omitted below and the gist will be described.

First, as shown in FIG. 16, the terminal 110 transmits a plurality of pieces of search data information including one or more search images input by a user to the server 120. A similar image information receiving step of receiving similar image information for a similar image searched by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information from 120; S220) and providing similar image information for providing the similar image information on the similar image to the user (S230).

At this time, in the search information transmission step (S210), a plurality of search attribute information for the plurality of search data information may be transmitted together with the plurality of search data information.

In addition, as shown in FIG. 17, the search information transmitting step (S210) may include a search image attribute for calculating a plurality of search image attribute feature vectors corresponding to the plurality of search attribute information in the one or more search images. The feature vector calculating step S211 and the feature vector transmission step S212 for transmitting the calculated feature vectors for each of the plurality of search image attributes to the server 120 may be included.

Further, in the transmitting of the feature vector for each search image property (S211), a plurality of search image property vectors corresponding to the plurality of search image properties calculated in correspondence with the plurality of search property information in the one or more search images are connected to one connected search image feature. A vector may be generated and transmitted to the server 120.

In addition, the computer program according to another aspect of the present invention is characterized in that the computer program stored in a computer readable medium for executing each step of the previously described image search method in a computer. The computer program may be a computer program including not only a computer program including machine code generated by a compiler, but also a high-level language code executable on a computer using an interpreter or the like. In this case, the computer is not limited to a personal computer (PC) or a notebook computer, but is provided with a central processing unit (CPU) such as a server, a smart phone, a tablet PC, a PDA, a mobile phone, and any information processing capable of executing a computer program. Device. In addition, the computer-readable medium may include an electronic recording medium (eg, ROM, flash memory, etc.), a magnetic storage medium (eg, floppy disk, hard disk, etc.), an optical reading medium (eg, CD-ROM, DVD, etc.), any storage medium that can be read by a computer.

In addition, FIG. 18 illustrates a configuration diagram of a server 120 that provides an image search according to an embodiment of the present invention.

As shown in FIG. 18, the server 120 for providing an image search according to an embodiment of the present invention includes a search information receiver 121, a similar image search unit 122, and a similar image information transmitter 123. Can be configured.

In the following, the server 120 for providing an image search according to an embodiment of the present invention is divided and searched for each component. For more information on the server 120 for providing an image search according to an embodiment of the present invention can be inferred from the description of the image search method according to an embodiment of the present invention described above, in more detail below Description is omitted.

First, the search information receiving unit 121 receives a plurality of pieces of search data information including one or more search images input by a user from the terminal 110.

In this case, the search information receiver 121 may receive a plurality of search attribute information for the plurality of search data information together with the plurality of search data information.

In addition, the similar image search unit 122 searches for similar images by combining search data partial information corresponding to respective search attribute information of the plurality of search data information in the plurality of search data information.

Finally, the similar image providing unit 123 transmits similar image information about the found similar image to the terminal.

In addition, FIG. 19 illustrates a configuration diagram of a terminal 110 that provides an image search according to an embodiment of the present invention.

As shown in FIG. 19, the terminal 110 providing an image search according to an embodiment of the present invention uses a search information transmitter 111, a similar image information receiver 112, and a similar image information provider 113. It can be configured to include.

In the following, the terminal 110 for providing an image search according to an embodiment of the present invention is divided and searched for each component. For more information on the terminal 110 providing an image search according to an embodiment of the present invention can be inferred from the description of the image search method according to an embodiment of the present invention described above, in more detail below Description is omitted.

First, the search information transmitter 111 transmits a plurality of search data information including one or more search images input by a user to the server 120.

In this case, the search information transmitter 111 may transmit a plurality of search attribute information for the plurality of search data information together with the plurality of search data information.

In addition, the similar image information receiving unit 112 combines the search data portion information corresponding to the respective search attribute information of the plurality of search data information from the plurality of search data information from the server 120 to a similar image searched for. Receive similar image information about an image.

Finally, the similar image information providing unit 113 provides similar image information on the similar image to the user.

Accordingly, in the image retrieval method, apparatus and computer program according to an embodiment of the present invention, in searching for and providing an image similar to a given search image, a similar image is searched and provided based on a specific property of the given search image. Further, by searching for and providing an image similar to the image combining the plurality of search images based on the respective search attributes of the plurality of search images, a similar image for the image combining the plurality of search images is more convenient. You will be provided correctly.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and changes without departing from the essential characteristics of the present invention. Therefore, the embodiments described in the present invention are not intended to limit the technical spirit of the present invention but to describe the present invention, and are not limited to these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: Image Search System
110, 110a, 110b: terminal
111: search information transmitter
112: similar image information receiving unit
113: similar image information providing unit
120: server
121: search information receiving unit
122: Similar Image Search Unit
123: Similar Image Information Transmitter
124: database
130: network

Claims (15)

A search information receiving step of receiving, by the server, a plurality of pieces of search data information including at least one search image input by a user from the terminal;
A similar image retrieval step of retrieving a similar image by combining retrieval data partial information corresponding to respective retrieval attribute information for the plurality of retrieval data information from the retrieval data information; And
A similar image providing step of transmitting similar image information about the retrieved similar image to the terminal;
Image retrieval method comprising a. The method of claim 1,
In the search information receiving step,
Receiving a plurality of search images input by the user and a plurality of search attribute information for the plurality of search images,
In the similar image search step,
And searching for a similar image by combining corresponding search image partial information in the plurality of search images for each of the plurality of search attribute information. The method of claim 2,
The similar image search step,
Calculating a feature vector for each search image property to calculate a feature vector for each search image property corresponding to the plurality of search property information from the plurality of search images; And
A database search step of searching for similar images by comparing feature vectors for each of the plurality of stored image attributes calculated and stored on the basis of the plurality of search attribute information with respect to the plurality of stored images stored in the database with respect to the feature vectors for each of the plurality of search image attributes ;
Image retrieval method comprising a. The method of claim 3,
The database stores a concatenated storage image feature vector to which the feature vector for each of the plurality of storage image attributes is connected to the plurality of storage images.
In the similar image search step,
After generating the connected search image feature vector to which the plurality of search image property feature vectors are connected
And searching for similar images based on similarity with the connected stored image feature vector stored in the database. The method of claim 3,
The calculating of the feature vector for each search image attribute may include:
Detecting at least one object included in the search image; And
Calculating a feature vector for each search image property for each of the one or more objects based on the search property information;
Image retrieval method comprising a. The method of claim 3,
In the feature vector calculation step for each search image attribute,
After indexing a plurality of elements included in the feature vector for each feature of the search image by size, an index feature having as its element an index value indicating a position in the feature vector for each search image of the largest k elements Yield a vector,
In the feature vector similar image search step,
And searching for similar images with respect to the searched image using the index feature vector. The method of claim 1,
In the search information receiving step,
Receiving search information including a first search image input by the user, first search attribute information thereof, second search text, and second search criteria information related thereto;
In the similar image search step,
The similar image is obtained by combining first search data portion information corresponding to the first search attribute information in the first search image and second search data portion information corresponding to the second search attribute information with respect to the second search text. Searching for an image, characterized in that for searching. The method of claim 1,
In the search information receiving step,
The terminal provides a user interface for selecting a search image and the search attribute information for the search image stored in advance,
In the similar image search step,
And searching for a similar image by using the previously stored search image and search attribute information thereof. The method of claim 1,
In the search information receiving step,
The terminal provides a user interface for setting the search attribute information for a plurality of search data information input by the user,
And receiving, from the terminal, search attribute information set for the plurality of pieces of search data information. A search information transmission step of transmitting, by the terminal, a plurality of search data information including one or more search images input by a user to a server;
A similar image information receiving step of receiving, from the server, similar image information for a similar image found by combining search data partial information corresponding to respective search attribute information for the plurality of search data information in the plurality of search data information; And
Providing similar image information for providing the user with similar image information on the similar image;
Image retrieval method comprising a. The method of claim 10,
The search information transmission step,
Calculating a feature vector for each search image property to calculate a feature vector for each search image property corresponding to the plurality of search property information in the one or more search images; And
Transmitting a feature vector for each search image property to transmit the calculated plurality of feature images for each search image property to the server;
Image retrieval method comprising a. The method of claim 11,
In the feature vector transmission step for each search image attribute,
An image retrieval method comprising generating a single connection search image feature vector by connecting a plurality of search image property feature vectors calculated in correspondence with the plurality of search property information in the one or more search images . A computer program stored in a computer readable medium for executing each step of the image retrieval method according to any one of claims 1 to 12 on a computer. A search information receiver configured to receive a plurality of search data information including at least one search image input by a user from a terminal;
A similar image search unit for searching for a similar image by combining search data portion information corresponding to each search attribute information of the plurality of search data information in the plurality of search data information; And
A similar image provider which transmits similar image information about the retrieved similar image to the terminal;
Server comprising a. A search information transmitter for transmitting a plurality of search data information including one or more search images input by a user to a server;
A similar image information receiver configured to receive similar image information for a similar image found by combining search data partial information corresponding to each search attribute information of the plurality of search data information in the plurality of search data information from the server; And
A similar image information providing unit providing similar image information on the similar image to the user;
Terminal comprising a.