JP2005107931A - Image search apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image search apparatus and an image search method that search many documents for an image and recognize an image from the document data. <P>SOLUTION: The image search apparatus comprises a registration processing means 11 for extracting text information indicating contents of images included in a part of documents from a text in the documents by natural language processing, and extracting feature quantities indicating the contents of the images from image data on the part of documents and the other part, a database 12 for saving the text information about the part of documents, the feature quantities about the part of documents and the other part, and the images, search processing means 13 for searching the text in the database in response to a search request from a user, and searching the feature quantities in the database according to the feature quantities of searched images, and a user interface 14 by which the user issues the search request and which presents the search result by the search processing means to the user. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image search apparatus and an image search method that search for images in a large amount of documents and recognize images based on the document data.

  Patent Document 1 relates to an image search device, a method for generating image search key text, a program for causing a computer to function as the device, and a computer-readable recording medium on which a program for causing the computer to execute the method is recorded. It is possible to search for an image in a document by text.

  Further, Patent Documents 2, 3, 4, and 5 disclose conventional techniques for performing image search using text data and image feature amounts.

JP 11-25113 A JP 2001-84274 A JP 2000-48041 A JP 2000-250943 A JP 2000-76287 A

  However, the search technique disclosed in Patent Document 1 has a problem that the search cannot be performed well when an appropriate text is not given to the image.

Further, in the techniques of Patent Document 2 to Patent Document 5, even when searching using both text and image feature amount, the search is performed using the text specified by the user and the image feature amount of the image. If the image feature amount is not similar, the search is not performed. If the text of the registered image is not set appropriately, the search is not performed unless the image feature amounts are similar.
Actually, even if the image feature amount is not similar, the semantic content of the image is often similar. In such a case, the techniques of Patent Documents 2 to 5 are not useful.

  It is an object of the present invention to enable searching even for images in which registered images are not properly set, and to search by text as a first step in response to a search request from a user, and text as a second step. The feature is to make it possible to search with the feature amount assigned to the image searched in step (b).

  The present inventor can extract the image intended by the user by performing a two-stage search process in response to a single search request from the user, and the text specified by the user and the user specified Knowledge that users can extract the image that the user intended by searching with the image feature amount of the image obtained as a result of searching with the text as a key, instead of simply searching with both the image feature amount of the acquired image as a key To obtain the present invention.

An image search apparatus according to a first aspect of the present invention searches for a predetermined image from a plurality of documents including images, and for some documents, text information indicating the contents of the images included in the document is Text information extraction means for extracting from a text in a document by natural language processing; and image feature quantity extraction means for extracting feature quantities indicating image contents from image data of the partial document and the other partial document; , A database for storing text information about the part document, the feature amount and the image about the part document and the other part document, and a user issuing a search request by text or keyword Search request issuance means, text search processing execution means for searching for text in the database in response to the search request, and feature amount of the searched image Image search processing execution means for searching for feature quantities from the database and searching for images of similar feature quantities, and search result presentation means for presenting the search results by the text search processing execution means and the image search processing execution means to the user It is characterized by comprising.
The image search apparatus of the present invention can be configured as a search system in which each component is arranged in a server and the server can be accessed by a mobile phone such as a terminal computer. Further, a stand-alone system using a DVD, CDROM, or the like as a database, or a system in which a database and a terminal computer are connected by a LAN can be used.

  In the image search device according to the first aspect, a plurality of texts searched by the text search processing execution unit and images associated therewith are displayed on the screen, and the user is allowed to select the image, and the selected image is displayed. Based on the above, the search processing execution means can search for the feature amount.

  In the image search device according to the first aspect, when a plurality of images are searched in the search by the text search processing execution means, the search result by the text search processing execution means can be displayed for each image.

  The image search apparatus according to the second aspect of the present invention is for searching for a predetermined image from a plurality of documents including images, and is text information indicating the content of an image from image data relating to the image included in the document A text information extracting means for extracting the image by natural language processing, and a feature for extracting a feature amount indicating the content of the image from image data relating to the image included in the document and extracting a feature amount for an arbitrary image input by the user Corresponding to the feature quantity retrieved by the feature quantity retrieval means, the database storing the text information, the feature quantity and the image, the feature quantity search means for retrieving a similar feature quantity from the database, and the feature quantity retrieval means And a search result presenting means for presenting text information to the user.

  In the image search device according to the second aspect of the present invention, the image search apparatus includes confirmation information display means for displaying an image searched by the feature amount search means and text information associated therewith on a screen. Check whether the search by the quantity matches the user's intention, or whether the image (further feature quantity) and the text are correctly associated with each other. Feature quantity / text correspondence release means for canceling the correspondence, and / or feature quantity / text correspondence confidence for setting the confidence level of the correspondence to the image (and the feature quantity) and the text, or changing the set value Degree setting means.

  In the image search apparatus according to the first and second aspects of the present invention, it is possible to map and search text for each language corresponding to a plurality of languages.

  The image search apparatus according to the first and second aspects of the present invention further includes automatic association means for automatically associating the text with the feature amount, and editing means for the user to perform the association. Can be provided.

  In the image search apparatus according to the first and second aspects of the present invention, in addition to the text and the feature amount, context information (Web page or document type) in which the text or feature amount appears is held, and the image search is performed. It is possible to search including the context information.

  The image search method according to the third aspect of the present invention is for searching a predetermined image from a plurality of documents including images, and for some documents, text information indicating the contents of the images included in the documents. A text information extraction step for extracting the text from the text in the document by natural language processing, and an image feature quantity extraction for extracting a feature quantity indicating the content of the image from the image data of the partial document and the other partial document Issued by a user, and a database storage step for storing text information about the part of the document and the feature quantity and the image of the part of the document and the other part of the document in a database A text search processing execution step for searching for text in the database in response to a search request by text or keyword, Based on the feature amount of the image, search the feature amount from the database and search for an image having a similar feature amount, and present the search results in the text search processing execution step and the image search processing execution step to the user And a search result presentation step.

  In the image search method of the third aspect of the present invention, the text searched in the text search processing execution step and a plurality of images associated therewith are displayed on the screen, and the user selects the image, Based on the selected image, the image of the feature amount can be searched in the image search processing execution step.

  In the image search method of the third aspect of the present invention, when a plurality of images are searched in the search in the text search process execution step, the search result in the text search process execution step can be displayed for each image.

  The image search method of the fourth aspect of the present invention is for searching for a predetermined image from a plurality of documents including images, and is text information indicating the contents of the image from image data relating to the images included in the document A text information extraction step for extracting the image by natural language processing, and extracting a feature amount indicating the content of the image from image data relating to the image included in the document, and extracting a feature amount for an arbitrary image input by the user The feature amount extraction step, the database storage step for storing the text information, the feature amount, and the image in a database, the feature amount search step for searching for a similar feature amount from the database, and the feature amount search step A search result presentation step for presenting text information corresponding to the feature amount to the user. And butterflies.

  The image search method according to the fourth aspect of the present invention includes a confirmation information display step for displaying on the screen the image searched in the feature amount search step and the text information associated therewith, and the image is operated by a user operation. Feature / text correspondence release step to cancel the association between text and text, and / or feature / text correspondence confidence setting to set the confidence level of the correspondence between images and text, or to change the set value Steps may be provided.

  In the image search methods of the third and fourth aspects of the present invention, it is possible to map and search text for each language corresponding to a plurality of languages and for one feature amount.

  The image search method according to the third and fourth aspects of the present invention can include an automatic association step of automatically associating the text with the feature amount.

  In the image search method according to the third and fourth aspects of the present invention, in addition to the text and the feature amount, the context information in which the text or the feature amount appears is held, and the context is searched including the context information when searching for the image. An information retrieval step can be provided.

(1) In the first and third aspects of the invention, an image search (first-stage image search) is performed using text as a key, and the next image search is performed using the image feature amount of the image obtained as a key. (Second stage image search) is performed. By performing such a search, it is sufficient that text information for search is included in only a part of registered images, and it is not necessary that text information is added to all registered images. In the conventional technique, when searching using text as a key, if the text is not correctly assigned to the image, the desired image cannot be obtained by the search. However, in the present invention, the text is correctly assigned. For this reason, even if the desired image cannot be acquired by the first-stage image search, the desired image can be acquired by the second-stage search if the feature amounts are similar. Thereby, omission of search can be eliminated and search accuracy can be improved.

(2) In the inventions of the first and third aspects, the search accuracy can be improved by allowing the user to select an appropriate image.

(3) In the invention of the first aspect and the third aspect, when a plurality of images are searched in the search in the first stage, the search result in the second stage for each image is displayed for each image. Thus, the user can easily grasp the search operation, and the designation at the next search becomes easy.

(4) In the second and fourth aspects of the invention, the user can acquire information such as text indicating the contents of the image by designating the image.

(5) In the inventions of the second aspect and the fourth aspect, the search accuracy can be improved by allowing the user to freely edit the association information during the search.

(6) In the first to fourth aspects of the invention, each language can be handled.

(7) In the inventions of the first to fourth aspects, it is possible for the user to improve the search accuracy by freely editing the association information.

(8) In the inventions of the first to fourth aspects, it is possible to increase the search accuracy by adding context information.

  Hereinafter, a search on the Internet, which is considered to be most effective by the image search apparatus of the present invention, will be described. All pages described in HTML on the Internet are documents (that is, composed of document data), and various images are embedded (that is, linked) in the document.

  FIG. 1 shows an embodiment of an image search apparatus of the present invention. In FIG. 1, the image search device 1 includes a registration processing unit 11, a database 12, a search processing unit 13, and a user interface 14, and the image search device 1 is connected to the Internet 100. The registration processing unit 11 functions as a text information extraction unit, a feature amount extraction unit, an automatic association unit, and the like according to the present invention. The search processing means 13 functions as a text search processing execution means, a search processing execution means, a confirmation information display means, a context information search means, etc. according to the present invention. The user interface 14 functions as a search request issuing unit and a search result presenting unit of the present invention, and functions as a confirmation information display unit, a feature / text correspondence release unit, a feature / text correspondence certainty setting unit, and the like.

  The text information extraction step, the image feature amount extraction step, the database storage step, the image search processing execution step, and the search result presentation step in the image search method of the present invention are text information extraction means, image feature amount extraction means, database, image This is realized by search processing execution means and search result presentation means. Also, the confirmation information display step, the feature quantity / text correspondence release step, and the feature quantity / text correspondence confidence setting step are realized by the confirmation information display means, the feature quantity / text correspondence release means, and the feature quantity / text correspondence confidence setting means. Is done. Furthermore, the automatic association step and the context information search step are realized by automatic association means and context information search means.

First embodiment

  Registration in the database in the first embodiment is performed as follows. In this embodiment, it is assumed that the HTML document to be registered is on the Internet or an intranet, but the present invention can also be applied when there is an HTML document on a local file or database. A document set designated in advance by a person may be registered. Here, an example in which a general crawler (robot) is used as an automatic page acquisition method on the Internet will be described. When searching, a text information search and a feature amount search are performed in two stages. The document searched by the text information may be the entire document or a part of the entire document if possible. Note that the image feature amount needs to be extracted from all search target documents in advance and registered in the database. FIG. 2 shows an outline of the registration process (S101 to S105), and FIG. 3 shows an outline of the search process (S201 to S204).

[Select registered images]
The crawler acquires one page (HTML document) existing on the Internet one by one (S101), and extracts an image from the acquired page (S102). That is, when there is a tag for designating an image, the image is registered in the database as a registered image (S103). When the extraction process ends or when the HTML document does not include an image, the next page is acquired and the extraction process is repeated.

  There are images (buttons, backgrounds, ruled lines, etc.) that are not targeted for image search in the images (linked images) included in the HTML. These images are excluded from the registered images as follows. That is, the registration target image is selected based on information such as the image size, aspect ratio, and type. For example, an image whose size is smaller than a predetermined size is assumed to be a button, a check box, or the like, and is not registered. In addition, an image with a large aspect ratio is not a registration target because it is assumed to be a ruled line. Furthermore, since GIF image data is often decorated, it is not a registered image.

[Text Information Extraction]
Text information indicating the contents of the image is extracted from the HTML document from the registered image (S104). In the case of an HTML document, information indicating the contents of the image is described in the alt attribute of the image (img) tag, and this is extracted as text information. For the extraction of text information, the method described in JP-A-11-25113 can be used.

  It is also possible to extract text information indicating the contents of the image. The text information may be a natural language text, or a keyword may be extracted using morphological analysis or the like. Also, a plurality of text information may be extracted and weighted for each text information as in the technique described in Japanese Patent Application Laid-Open No. 11-25113.

  The extracted text information is registered in the database. Extraction of text information does not have to be performed from all of the registered images, and if an appropriate text cannot be extracted, only feature amount extraction described later is performed. If the text information is already registered as a result of extracting the text information and there are many images related to the text information, the text information may or may not be registered in the database. .

[Image feature extraction]
Image feature amounts are extracted from the image data to be registered (S105). A general image feature amount such as a color histogram can be used as the image feature amount. For example, in the case of a color histogram, the feature amount is vector data. Further, instead of extracting the image feature amount from the entire image, the image feature amount may be extracted from a region (object) extracted by region identification. The extracted image feature quantity and image data are associated and registered in the database. The URL of the image may be registered without registering the image data itself. Unlike text information, image feature amount extraction needs to be extracted from all registration target images.

  Hereinafter, the text image search will be described. The user can search for a desired image by text. In this search, an image search (first-stage image search) is first performed using text as a key in response to a single search request from the user, and an image search based on the feature amount of the image searched in the first stage. (Second-stage image search) is performed, and the search results obtained by the two-stage search are presented to the user. In the second stage search, the search is performed using the feature amount linked to the image searched in the first stage as a key.

[Text Information Search]
Natural language text or keywords (search text) input from the user interface 14 are acquired, and in the case of natural language texts, keywords are extracted by morphological analysis (S201). Search text information in database by keyword. That is, text information including a keyword is searched by full text search, and an image feature amount corresponding to the searched text information is specified. In some cases, text information of a plurality of images is searched, and a plurality of image feature amounts are specified. In addition, when certainty factors are specified, they are sorted in the order of certainty factors, and are limited to a plurality of images specified in advance or limited to images having a predetermined threshold value or more. Is identified.

  It is also possible to list images from which image feature amounts specified by text are extracted, and the user can explicitly specify an image similar to an image intended by the user. FIG. 4 shows an example of this display screen. In addition to selecting an image for search on the screen as shown in FIG. 4, it is possible for the user to add a check box for designating that the combination of the image and the text is not correct. When the user designates the check box, the association between the image and the text is changed more accurately by deleting (disconnecting) the association between the image and the text or by reducing the certainty level. It becomes possible.

[Image feature search]
An image feature quantity similar to the inquiry image feature quantity specified by the text information search is searched from the database (S202). If the image feature quantity is vector data such as a color histogram, the similarity can be calculated by, for example, the Euclidean distance. The similarity between the inquiry image feature quantity and all the feature quantities in the database is calculated, and the feature quantities are sorted in descending order of similarity. When a plurality of image feature amounts are specified in the text information search, the similarity can be the sum of the similarities of the feature amounts, and the minimum similarity can be used instead of the sum. Good.

[Search result display]
Images corresponding to the order of the feature quantities sorted in the image feature quantity search are displayed to the user (S203). The user can select an image from the displayed image list and acquire data of the image.
When an image is selected, it is possible not only to list searched images but also to display an image obtained as a result of searching for each selected image as shown in FIG.

[Confidence Update]
In the search result display process, the fact that the user selects an image means that the text specified by the user matches the image. Accordingly, since it can be determined that the combination of the image feature amount and the text information specified by the text information search is appropriate, the certainty factor for the combination of the image feature amount and the text information registered in the database is updated (S204). . By doing so, a search with a high certainty factor is preferentially searched, so that the search accuracy can be improved.

[Multi-language support]
Text information managed in the database can be stored for each language. That is, each image has text information of Japanese, English, and French, and the search can be performed in any language. It is also possible to search the feature amount of the image based on the image searched in Japanese and to search English associated with the searched feature amount. That is, English corresponding to Japanese can be searched, and can be used like a Japanese-English dictionary.

[Edit feature-text association]
The accuracy of the search greatly depends on the information relating the feature quantity and the text. This association information is not only automatically extracted, but can be explicitly edited by the user. You can simply list the associations and delete the associations or change the confidence of the associations.

[Search by context information]
It is also possible to add context information to the information on the association between the text and the feature amount. The context information is information indicating what type of page the text or image appears on. For Web pages and documents, the text type can be extracted from the title and text information of the page. For example, it is possible to extract keywords as a result of morphological analysis of text in the page, and if there are many engineering-related keywords as a result, it is determined that the page is engineering-related, and a value indicating engineering can be assigned to the association information as context information. is there. When searching, it is possible to select not only the text but also the context, and the search accuracy can be further improved by searching the association information together with the designated context.

Second embodiment

  An image can be recognized with the same configuration as in the first embodiment, and text indicating the content of the image can be displayed. Since the database of the first embodiment can be used as it is as a recognition database, it can be used together with the first embodiment. Since registration is the same as in the first embodiment, description thereof is omitted. However, when only the recognition is intended, it is not necessary to register all images, and it is necessary to always extract both text information and feature amounts. An outline of the search process is shown in FIG. 6 (S301 to S303). The user can specify an arbitrary image to be recognized and obtain text indicating the content of the image.

[Image feature search]
Image feature amounts are extracted from the designated image, and similar images and feature amounts are searched from the database as in the first embodiment (S301). Before extracting the image feature quantity, the area may be extracted to extract the area (object), and then the image feature quantity may be extracted from the area. When the certainty factor is designated, the similarity is weighted by the certainty factor during the search.

[Search result display]
A list of text information corresponding to the image feature amount whose similarity is equal to or greater than a specific threshold value or a predetermined number of cases is presented to the user (S302).

[Confidence Update]
As in the first embodiment, the user can select an appropriate text and updates the certainty factor corresponding to the selected text information (S303).

[Multi-language support]
As in the first embodiment, text data can have a plurality of languages, and search results can be displayed in a designated language.

[Edit feature-text association]
The accuracy of the search greatly depends on the information relating the feature quantity and the text. This association information is not only automatically extracted, but can be explicitly edited by the user. You can simply list the associations and delete the associations or change the confidence of the associations.

[Search by context information]
It is also possible to add context information to the information on the association between the text and the feature amount. The context information is information indicating what type of page the text or image appears on. For Web pages and documents, the text type can be extracted from the title and text information of the page. For example, it is possible to extract keywords as a result of morphological analysis of text in the page, and if there are many engineering-related keywords as a result, it is determined that the page is engineering-related, and a value indicating engineering can be assigned to the association information as context information. is there. When searching, it is possible to select not only the text but also the context, and the search accuracy can be further improved by searching the association information together with the designated context.

It is a block diagram concerning 1st Embodiment of the image search device of this invention. It is a flowchart which shows the registration process in embodiment of FIG. It is a flowchart which shows the search process in embodiment of FIG. It is a figure which shows the example of a display of the search result in 1st Embodiment. It is a figure which shows the other example of a display of the search result in 1st Embodiment. It is a flowchart which shows the search process in 2nd Embodiment of the image search device of this invention.

Explanation of symbols

11 Registration Processing Unit 12 Database 13 Search Processing Unit 100 Internet

Claims (16)

An image search device for searching for a predetermined image from a plurality of documents including images,
Text information extracting means for extracting text information indicating the content of an image included in the document from the text in the document by natural language processing for some documents;
Image feature amount extraction means for extracting feature amounts indicating image contents from image data of the partial document and the other partial document;
A database for storing text information about the part document, and the feature amount and the image about the part document and the other part document;
A search request issuing means in which a user issues a search request by text or keyword;
Text search processing execution means for searching for text in the database in response to a search request;
Image search processing execution means for searching for a feature quantity from the database and searching for an image with a similar feature quantity based on the feature quantity of the searched image;
Search result presenting means for presenting a search result by the text search processing executing means and the image search processing executing means to the user;
An image search apparatus comprising:   The text searched by the text search processing execution means and a plurality of images associated therewith are displayed on the screen, the user is allowed to select the image, and the image search processing execution means is based on the selected image. The image search apparatus according to claim 1, wherein an image having the feature amount is searched.   3. The image search apparatus according to claim 1, wherein when a plurality of images are searched in the search by the text search process execution unit, a search result by the text search process execution unit is displayed for each image. . An image search device for searching for a predetermined image from a plurality of documents including images,
Text information extracting means for extracting, by natural language processing, text information indicating the content of an image from image data relating to an image included in the document;
A feature amount extracting unit that extracts a feature amount indicating the content of an image from image data relating to an image included in the document and extracts a feature amount for an arbitrary image input by a user;
A database for storing the text information, the feature amount, and the image;
Feature quantity search means for searching for similar feature quantities from the database;
Search result presentation means for presenting text information corresponding to the feature quantity searched by the feature quantity search means to the user;
An image search apparatus comprising: A confirmation information display means for displaying an image searched by the feature amount search means and text information associated therewith on a screen;
Feature amount / text correspondence canceling means for canceling the correspondence between the image and the text by the user's operation, and / or
A feature amount / text correspondence certainty setting means for setting the certainty of the correspondence between the image and text, or changing the set value,
The image search apparatus according to claim 4, further comprising:   6. The image search apparatus according to claim 1, wherein the image search apparatus corresponds to a plurality of languages and maps and searches text for each language with respect to one feature amount.   7. The apparatus according to claim 1, further comprising automatic association means for automatically associating the text with the feature quantity, and further comprising editing means for a user to perform the association. Image search device.   2. The apparatus according to claim 1, further comprising: context information search means for holding context information in which the text or feature amount appears in addition to the text and the feature amount, and for searching including the context information when searching for an image. 8. The image search device according to any one of 7 above. An image retrieval method for retrieving a predetermined image from a plurality of documents including images,
A text information extraction step for extracting text information indicating the content of an image included in the document from a text in the document by natural language processing for some documents;
An image feature amount extraction step for extracting a feature amount indicating the content of the image from the image data of the partial document and the other partial document;
A database storage step of storing the text information about the part of the document and the feature amount and the image of the part of the document and the other part of the document in a database;
A text search processing execution step for searching text in the database in response to a search request by text or a keyword issued by a user;
An image search processing execution step of searching for a feature amount from the database and searching for an image with a similar feature amount based on the feature amount of the searched image;
A search result presenting step for presenting search results to the user in the text search processing execution step and the image search processing execution step;
An image search method characterized by comprising:   The text searched in the text search processing execution step and a plurality of images associated therewith are displayed on the screen, and the user is allowed to select the image, and based on the selected image, in the image search processing execution step The image search method according to claim 9, wherein an image having the feature amount is searched.   The image search method according to claim 9 or 10, wherein when a plurality of images are searched in the search in the text search processing execution step, a search result in the text search processing execution step is displayed for each image. . An image retrieval method for retrieving a predetermined image from a plurality of documents including images,
A text information extraction step of extracting text information indicating the content of the image from image data concerning the image included in the document by natural language processing;
A feature amount extraction step of extracting a feature amount indicating the content of the image from image data of the image included in the document and extracting a feature amount for an arbitrary image input by the user;
A database storage step of storing the text information, the feature amount, and the image in a database;
A feature amount search step of searching for a similar feature amount from the database;
A search result presentation step of presenting text information corresponding to the feature amount searched in the feature amount search step to the user;
An image search method characterized by comprising: A confirmation information display step for displaying on the screen the image searched in the feature amount search step and the text information associated therewith,
A feature / text correspondence release step for releasing the correspondence between the image and the text by the user's operation, and / or
A feature / text correspondence confidence setting step for setting the certainty of the correspondence between the image and the text or changing the set value,
The image search method according to claim 12, further comprising:   The image search method according to any one of claims 9 to 13, wherein text is mapped for each language corresponding to a plurality of languages and searched for each language.   The image search method according to claim 9, further comprising an automatic association step of automatically associating the text with the feature amount.   10. A context information search step for storing context information in which the text or feature amount appears in addition to the text and the feature amount, and searching for the image information including the context information at the time of image search. The image search method according to any one of 15.

Images