JP2000048041A - Data retrieval system and device to be used for the system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make a system adaptive to the sensitivity of a user without making the user feel burden by automatically extracting a keyword or feature amount from registered data and narrowing down the retrieved result without using any keyword. SOLUTION: The feature amount weight by keyword of a feature amount weight block 101 by keywords is calculated, the keyword/feature amount relation strength value of a register image evaluation block 102 is calculated and an image is registered. The registered image is retrieved by a retrieval block 103 and the keyword used for keyword retrieval and the image designated by narrowing down the retrieved result are fed back to the feature amount weight block 101 by keywords and register image evaluation block 102. The feature amount weight block 101 by keywords adds the image designated by narrowing down the retrieved result to a sample image and the feature amount weight by keywords is calculated again. In the register image evaluation block 102, the image feature amount of the registered image is read out of a register image database and the keyword/feature amount relation strength value is calculated again.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data retrieval system capable of retrieving an image desired by a user using a keyword, and more particularly to data for automatically assigning a keyword to data registered in the data retrieval system. Learning about the relationship between keywords and keywords, learning the characteristics of desired data to narrow the search results by keyword to only desired data, and learning that reflects the results of searching by keywords and narrowing down to the desired data in the relationship between data and keywords is there.

[0002]

2. Description of the Related Art In recent years, there has been a demand for a data search, that is, an image search system capable of efficiently obtaining image data desired by a user from a large amount of stored data (here, represented by image data). Conventionally, a problem with the image search system is that it is not efficient because the task of assigning a keyword to an image is performed manually. However, the technology for automatically extracting the contents of an image has not yet reached a level where it can be put to practical use. Therefore, it is not a keyword that expresses what is in the image,
2. Description of the Related Art An image search system that automatically extracts, as a keyword, a sensuous expression for expressing a feeling received from an image (hereinafter, referred to as a “sensitive word”) has attracted attention. However, the sensibility is subjective and the system needs to be adjusted by the user.

In order to overcome such a problem, for example, (1) an image for automatically adding a kansei word to an image by extracting a rule for associating a kansei word with an image feature using a sample image. Search system (Japanese Unexamined Patent Publication No.
9351) and considering that the sensibility varies from user to user, (2) the user evaluates the image presented as a search result using the kansei word, and determines the difference in kansei for the kansei word in the neural network. A device for learning as a difference in weight and correcting an individual difference for each sensibility word (Research Society of Image Electronics Engineers, 96-02-02) is known.

The above (1) will be described with reference to FIG. In order to construct the image search system of the above (1), it is necessary to find a mapping rule f for converting a physical feature that can be mechanically extracted into a kansei feature used in an actual search. In the example shown in the figure, conversion of density values, elimination of noise,
Extraction of various physical features such as the number of boundary pixels, curve degree, complexity, number of colors, color distribution, contrast, etc. by applying various image processing such as restoration of blur, enhancement of contour, extraction of connected parts I do. In addition, the emotional features of the same 50 sample images are extracted by conducting a questionnaire survey on a plurality of subjects. The degree of sharp softness, simple decorativeness, dynamic / static, modern classical, abstract /
Sensitive features such as the degree of realism are answered at points of 0 to 1, and the sensible features that a human perceives for each sample image are digitized.

[0005] Once the sensual feature and the physical feature have been extracted from each of the images constituting the sample analysis image in this manner, the mapping rule f for defining the correlation between the two features is then obtained. Is created by a statistical method. Since the mapping rule f defines the correlation between the two feature values, the emotion feature value can be predicted if the physical feature value of a certain image is known.

On the other hand, a large number of images (1000 images are shown in FIG. 9) registered in the image database of the image search system are image-processed by a computer, and physical features of the same item as the sample image are obtained. Extract mechanically. For all registered images, sensual features are automatically generated automatically from physical features of the registered images using mapping rules f generated from analysis of the sample images. The perceptual features that humans perceive for an image must be evaluated one by one for each image as performed in the analysis of the sample image. The quantity can be determined.

[0007] When a registered image is registered in an image database in association with a kansei feature amount and an operator searches for a desired image using kansei words, a kansei that matches or approximates the inputted kansei words. The registered image having the characteristic feature is searched. This makes it possible to easily search for an image that is close to what the operator subjectively desires, as compared to performing an image search using physical feature values.

The above (2) will be described with reference to FIG. FIG. 10 is a flowchart illustrating a procedure of a process in which a user performs an image search using a sensitivity word. In this case, when the sentiment word is input, in processing step (hereinafter simply referred to as step) 1, weight data is read from the weight database, network weights are initialized, and then feature amount data is read from the feature amount database. The candidate images are read out and determined by the neural network, and a plurality of candidate images are output as a search result (step 2). As a result, the image data is read from the image database 13, and it is checked whether or not the user is satisfied with the image data (step 3). In this check process, if there is a favorite image among the candidate images, writing of weight data is performed (step 4), and further image acquisition is performed (step 5), and the search process ends. On the other hand, if it is determined in step 3 that there is no favorite image, the user enters candidate image evaluation processing, and the presented image is evaluated for the sentiment words used in the search in seven levels (step 6). ). The result of the seven-step evaluation is generated and input as a teacher signal to the neural network shown in FIG. 11 (step 7), and learning of the neural network is performed using the input as the feature amount of the candidate image (step 8). Return and a new candidate image is presented. By repeatedly performing the evaluation in this manner, the possibility that an image suitable for the kansei word of the person who performed the evaluation is increased.

[0009]

However, in the above method (1), once a mapping rule is generated using a sample image, the mapping rule is not updated. Can not respond to the change of the image. Further, there is a problem that it is impossible to cope with a difference in sensitivity of each user. Further, in the above (2), the user needs to repeatedly give an evaluation in order to deal with the user, so that the burden on the user is large. Further, there is a problem that it is not possible to cope with a search by a combination of keywords.

According to the present invention, in an image retrieval system using a sentiment word as a keyword, the load of image registration is reduced by automatically extracting the sentiment word from the image, and the retrieval by the combination of the sentiment words and the direct search of the image are performed. By combining search refinements to be specified, a user's intuitive search can be realized, and information generated during the search refinement operation can be used to update the learning to match the user's sensitivity. It is an object of the present invention to be able to perform an image search that can be adapted to the user's sensitivity without causing the user to feel a burden.

[0011]

SUMMARY OF THE INVENTION In order to solve this problem, the present invention provides a data retrieval system for retrieving data by using a keyword, a keyword inputting means for inputting a keyword, a data inputting means for inputting data, and a registration method. A data storage unit for storing a data feature amount and a keyword together with data; a keyword search unit for searching for data by combining a keyword input from the keyword input unit; and a feature for automatically extracting a keyword or a feature amount from registered data. It is provided with an amount / keyword extracting means and a search result narrowing means for narrowing down a search result by the keyword searching means without using a keyword.

[0012] The search result narrowing means includes a display means for displaying a part or all of the search results, and a data for designating data similar to desired data from the data displayed on the search result display means. Specifying means, learning means for learning the characteristics of the desired data, using the data data displayed by the search result display means as learning data, classification in accordance with the characteristics of the desired data, and outputting the classification result to the display means Classifying means, and the feature amount / keyword extracting means, keyword / feature amount relationship strength calculating means for calculating the strength of the relationship between the keyword and the feature amount using the registered data data and the keyword in the data storage unit; A part or all of the search results narrowed down by the search result narrowing means, the keyword specified in the keyword search, and the registration in the data storage unit are registered. Keyword / feature relation strength recalculation means for recalculating the strength of the relationship between the keyword and the feature quantity using the data data, data selection means for selecting data stored in the data storage means, and data from the data storage means. A data reading means for reading the data, a data feature calculation means for calculating a feature of the data read by the data reading means, a feature calculated by the data feature calculation and the keyword / feature relation strength calculation means, A calculation result writing unit that outputs the keyword / feature value relationship strength value calculated by the keyword / feature value relationship strength recalculation unit to the data storage unit.

According to the present invention, in a data search system using a sentiment word as a keyword, a sentiment word is automatically extracted from data, thereby reducing the data registration load. By combining search refinements that do not use words, a user's intuitive search can be realized, and the user can be naturally adapted during the search operation.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS According to a first aspect of the present invention, there is provided a data retrieval system for retrieving data by using a keyword, a keyword inputting means for inputting a keyword, a data inputting means for inputting data, and a registration data. Data storage means for storing a data feature amount and a keyword, keyword search means for searching for data by combining the keyword input from the keyword input means, and a feature quantity for automatically extracting a keyword or a feature amount from registered data -It is characterized by comprising a keyword extracting means and a search result narrowing means for narrowing down a search result by the keyword searching means without using a keyword, and learning a correspondence relationship between a keyword and a data feature amount (keyword automatic extraction). To be able to reflect search results To.

According to a second aspect of the present invention, there is provided an image search system for searching for an image by using a keyword,
Keyword inputting means for inputting a keyword, image inputting means for inputting an image, image storing means for storing an image feature amount and a keyword together with a registered image, and searching for an image by combining the keyword input from the keyword inputting means Keyword search means, feature quantity / keyword extraction means for automatically extracting keywords and feature quantities from registered images, and search result narrowing means for narrowing down the search results obtained by the keyword search means without using keywords. This has the effect that the narrowed search results can be reflected in the learning of the correspondence between keywords and image feature amounts (keyword automatic extraction).

According to a third aspect of the present invention, in the image search system according to the second aspect, the search result narrowing means comprises: a display means for displaying a part or all of the images of the search result; Image designating means for designating an image similar to the desired image from the images displayed on the result display means, and learning means for learning features of the desired image using the image data displayed on the search result display means as learning data And a classification means for classifying according to the characteristics of the desired image and outputting a classification result to the display means. Even if an appropriate keyword cannot be conceived, the search is narrowed down intuitively. It has the effect of being able to.

According to a fourth aspect of the present invention, in the image retrieval system according to the second or third aspect, the feature / keyword extracting means uses the image data and the keyword registered in the image storage unit. Keyword / feature relationship strength calculation means for calculating the strength of the relationship between the keyword and the feature quantity, registration of part or all of the search results narrowed down by the search result narrowing means, the keyword specified in the keyword search, and the image storage unit Keyword / feature relation strength recalculation means for recalculating the strength of the relationship between the keyword and the feature quantity using the obtained image data, image selection means for selecting an image stored in the image storage means, and image storage means Image reading means for reading an image from an image, image feature amount calculating means for calculating a feature amount of the image read by the image reading means, and the image feature A calculation result of outputting the feature strength calculated by the calculation means and the strength of the relationship between the keyword and the feature quantity calculated by the keyword / feature quantity relationship strength calculation means and the keyword / feature quantity relationship strength recalculation means to the image storage means. It is characterized by having a writing means, and has an effect that the automatic keyword extraction can be adjusted according to the user and the adjustment according to the addition / deletion of the image can be performed.

According to a fifth aspect of the present invention, in the image retrieval system according to any one of the second to fourth aspects, the image storage means has a strong relationship between the image and all keywords separately from the keywords. This feature is characterized by storing a value representing the value, and has an effect that the correlation between the keyword and the image not assigned to the image and the image can be determined.

According to a sixth aspect of the present invention, there is provided an image feature for selecting and reading a sample image from a registered image database for storing a registered image and a registered image database and extracting an image feature amount of the sample image. Amount extraction means, keyword input means for adding a keyword to a sample image or correcting a keyword already added, similarity between images with the same keyword increased, and similarity with an image without the same keyword And a keyword-based feature amount weight calculating means for calculating a feature amount weight such that the degree is reduced for each keyword.The registered image is learned by calculating a feature amount weight for each keyword, and the feature of the registered image is further improved. This has the effect of making the meaning closer to the meaning of the keyword.

According to a seventh aspect of the present invention, there is provided a registered image database for storing a registered image, an image characteristic amount extracting means for extracting an image characteristic amount from registered image data, a keyword and a registered image characteristic amount. A keyword / feature amount relationship strength calculating means for calculating a relationship strength value, wherein for a registered image, the relationship strength between the keyword and the registered image feature value is determined to facilitate setting or updating of the keyword. Having.

According to an eighth aspect of the present invention, there is provided an image feature for selecting and reading a sample image from a registered image database for storing a registered image and a registered image database and extracting an image feature amount of the sample image. Amount extraction means, keyword input means for adding a keyword to a sample image or correcting a keyword already added, similarity between images with the same keyword increased, and similarity with an image without the same keyword A feature amount weight learning unit including keyword-based feature amount weight calculation unit for calculating a feature amount weight for each keyword, and a registered image database storing a registered image; and an image feature amount from the registered image data. An image feature extracting means for extracting, and a key for calculating a relationship strength value between a keyword and a registered image feature. Word / feature value relationship strength calculation means; and registered image evaluation means for calculating the relationship strength between the keyword and the registered image feature value for the registered image, wherein the feature value weight output from the feature value weight learning means is provided. It has the effect of inputting the data to the keyword / feature relation strength calculating means of the registered image evaluation means and performing arithmetic processing to set and update the keyword for the registered image.

According to a ninth aspect of the present invention, there is provided an image feature for selecting and reading a sample image from a registered image database storing a registered image and a registered image database and extracting an image feature amount of the sample image. Amount extraction means, keyword input means for adding a keyword to a sample image or correcting a keyword already added, similarity between images with the same keyword increased, and similarity with an image without the same keyword A feature amount weight learning unit including keyword-based feature amount weight calculation unit for calculating a feature amount weight for each keyword, and a registered image database storing a registered image; and an image feature amount from the registered image data. An image feature extracting means for extracting, and a key for calculating a relationship strength value between a keyword and a registered image feature. Word / feature value relationship strength calculation means; registered image evaluation means for calculating the relationship strength between the keyword and the registered image feature value for the registered image; feature value weight data output from the feature value weight learning means for the registered image A keyword setting device configured to input to the keyword / feature relation strength calculating means of the evaluation means and perform arithmetic processing to set and update a keyword for a registered image; and a keyword input means for performing a keyword search; A keyword search unit for searching for a registered image corresponding to the keyword from the registered images; an image designating unit for specifying a part of an image to be narrowed down from search results of the keyword search unit; and a search result of the keyword search unit. An image search device including a search result narrowing unit for narrowing down and a search result display unit for displaying search results is provided. Is intended been example, have the effect that held the keyword setting function in the image retrieval system.

According to a tenth aspect of the present invention, in the keyword setting device according to the eighth aspect, a keyword
The feature amount relation strength calculation means rearranges the keywords in descending order of the keyword / feature amount relation strength value, and performs a process of adding the top N keywords to each image. It has the effect of being able to set.

According to an eleventh aspect of the present invention, in the keyword setting device according to the eighth aspect, a keyword
In the feature value relation strength calculation means, the keyword / feature value relationship strength is used as the weight of the keyword in the registered image,
The distance between the registered image and the learning image is measured in the image feature space by the K-NN method, and the top K images with the shortest distance are selected, and the keywords assigned to the selected K images are checked. A process of giving the top N frequently-used keywords to an image is performed.
Even if the data is unknown, the class to which the keyword belongs can be easily determined by using the learning data, and the keyword can be assigned to the image.

According to a tenth aspect of the present invention, in the keyword setting device according to the eighth aspect, a product of a keyword weight and a use frequency is treated as a weighted use frequency, and the product is calculated in an image feature space by a K-NN method. The distance between the registered image and the learning image is measured, and the top K of the images having the shortest distance are selected, and the keywords assigned to the selected K images are checked. The process to add to the image is performed, and even if the data is unknown, the optimal class to which the keyword belongs can be easily determined using the learning data,
This has an effect that a keyword can be assigned to an image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1) FIG. 1 is a system configuration diagram for describing an example of an image search system as a data search system according to Embodiment 1 of the present invention.
The image retrieval system according to the present embodiment includes a feature amount weight learning block 101 for each keyword using a sample image, and a relationship between a feature amount of a registered image using a sample image, a feature amount weight for each keyword, and a registration image and a keyword. A registered image evaluation block 102 for calculating an intensity value, a search block 103 for performing a search by a keyword and a search refinement by specifying an image, a registered image to which an image feature amount and a keyword / feature amount relationship strength value are added, And a registered image database 104 including feature weights.

Keyword-specific feature weight learning block 10
1 is a sample image 111 selected from registered images.
And an image feature amount extraction unit 112 for extracting an image feature amount of the sample image 111, a keyword input unit 132 for adding a keyword to the sample image 111 or correcting a keyword already added, The image processing apparatus further includes a keyword-specific feature weight calculation unit 113 that calculates a feature weight for each keyword such that the similarity between images increases and the similarity to an image without the same keyword decreases.

The registered image evaluation block 102 includes an image feature amount extraction unit 122 for extracting the image feature amount of the registered image data 121 and a keyword / feature amount relationship strength for calculating a relationship strength value 124 between the keyword and the registered image feature amount. Calculation unit 1
23.

The search block 103 includes a keyword input unit 132 for performing a keyword search, a keyword search unit 131 for searching a registered image corresponding to a keyword from registered images, and a search result of the keyword search unit 131. Image specifying unit 1 for specifying a part of an image to be narrowed down
34, a search result narrowing unit 133 for further narrowing down the search results of the keyword search unit 131, and a search result display unit 135 for displaying the search results.

In order to construct the above-described image retrieval system, first, a keyword-specific feature weight of the keyword-specific feature weight block 101 is calculated, and then a keyword-feature-value relationship strength value of the registered image evaluation block 102 is calculated. Is calculated and the image is registered. A search for a registered image is performed in a search block 103, and as a search result, the keyword used for the keyword search and the image specified by narrowing down the search result are fed back to the keyword-specific feature amount weight block 101 and the registered image evaluation block 102. In the keyword-specific feature weight block 101, the image specified by the search result narrowing is added to the sample image, and the keyword-specific feature weight is recalculated so that the similarity with the keyword used in the keyword search is improved. . In the registered image evaluation block 102, the image feature amount of the registered image is read from the registered image database, and the keyword / feature amount relationship strength value is recalculated.

In the following, the calculation of the keyword-specific feature weight, the calculation of the keyword / feature relationship strength value, the keyword search and the search result narrowing down, the recalculation of the keyword-specific feature weight, and the recalculation of the keyword / feature relationship strength value will be specifically described. Will be described.

FIG. 2 shows a processing flow of the feature amount weight calculation for each keyword. First, a keyword K for calculating the feature weight is designated (step 101). Next, among the registered images, at least two images with the keyword K are added.
A plurality of sample images are selected so as to include one or more (step 102). It is determined whether or not the keyword attached to the selected sample image is an appropriate keyword (step 103).
32 and input an appropriate keyword (step 1).
04). If the keyword of the sample image selected in step 103 is an appropriate keyword or if a keyword is input, step 1
It is determined whether or not the feature amount of the sample image selected in 02 has been extracted (step 105). If the feature amount has not been extracted, the feature amount is extracted by the image feature amount extraction unit 112 (step 106). Image feature extraction unit 112
Then, for example, feature amounts as shown in Table 1 are extracted and recorded in the registered image database 104. The normalized values are used for the feature amounts shown in Table 1.

[0034]

Next, the sample images are classified into an image group A having the keyword K and an image group B having no keyword K (step 107). Hereinafter, the feature amount weight WK (t) of the keyword K is calculated using the image group A and the image group B. The initial time is set to t = 0, and the feature amount weight vector WK (t) is initialized (step 10).
8) A random value is used as the initial value. It is determined whether or not the end condition is satisfied (step 109). If the end condition is satisfied, the process ends. If the end condition is not satisfied, two images are randomly selected from image group A. (Step 110). Step 1
The difference square vector d1 (t) of the feature amount of the image selected in 10 is selected by (Equation 1) (step 111).

[0036]

(Equation 1)

Here, the number of feature values is set to 2 for easy understanding. The feature values of the two images are defined as (a
1, a2), (b1, b2), 1 out of image group A
First, one image is randomly selected one by one from the image group B (step 112). As in the case of d1 (t), a difference square vector d2 (t) between the two images is calculated (step 113). When the feature quantity weight vector WK (t + 1) is expressed by an equation, the following (Equation 2) is calculated (step 114).

[0038]

(Equation 2)

Next, the process returns to step 109 with t = t + 1 (step 115). As described above, by repeatedly performing the operations from step 109 to step 115,
The weights are updated so that the distance between the images of group A becomes smaller and the distance between the images of group A and group B becomes relatively larger. When the number of feature amounts is 2, the image feature amounts are (a1, a2), (b1, b2), and the weight vector is (w1, w2), the distance between the two images is obtained by (Equation 3). .

[0040]

(Equation 3)

As a termination condition, a dispersion ratio is used, and a time t at which the dispersion ratio reaches a peak is detected. The variance ratio is a ratio between the variance between classes and the variance of all cases, and indicates that the larger the value of the variance ratio, the easier the classes are separated. The classes here correspond to two classes, group A and group B. The variance ratio is obtained by (Equation 4), the variance between classes is obtained by (Equation 5), and the variance of all cases is obtained by (Equation 6).

[0042]

(Equation 4) The number of attributes, the number of images in group A, the number of images in group B, the attribute of the i-th image, the average of the attributes of group A, the average of the attributes of images in group B, the average of all sample images Then

[0043]

(Equation 5)

[0044]

(Equation 6)

FIG. 3 is a flow chart showing the operation of calculating the keyword / feature amount relationship strength value. First, the user specifies which image is to be registered (step 201). It is detected whether the feature of the designated image has been extracted (step 20).
2) If the feature has not been extracted, the feature is extracted by the image feature extractor 122 (step 203). The feature amount extraction process performs the same process as step 106. Further, the image feature amount extraction unit 122 and the image feature amount extraction unit 112
Are exactly the same functional blocks. Next, it is determined whether or not the keyword / feature amount relationship strength values of the registered image have been calculated for all keywords (step 204). If all the keywords have been calculated, the process ends. If there is a keyword / feature value relationship strength value that has not been calculated, the keyword / feature value relationship strength value is calculated in the next step (step 205) and thereafter. First, a keyword K to be calculated is selected (step 205). Next, the feature amount weight WK of the corresponding keyword calculated in steps 101 to 115 is read from the registered image database 104 (step 206). The feature amount of the sample image similar to the sample image selected in step 102 is registered in the registration database 10.
4 is read (step 207). In the same manner as in step 107, a sample image is divided into an image group A having the keyword K and an image group B having no keyword K.
(Step 208). The image feature amount of the sample image is weighted by the weight WK (step 209), and similarly, the image feature amount of the registered image is weighted by the weight WK (step 210). The distance between the image of the image group A and the image of the image group B is calculated, and the maximum value is set as Dmax (step 211). The method of calculating the distance is selected according to (Equation 3). Further, the distance between the image of the image group A and the registered image is calculated, and the minimum value is set as Rmin (step 212).
Next, the distance between the sample image and the registered image is calculated, and the top N sample images having a short distance are selected (step 212).
a). The number of images of the group A contained in the top N sample images and x _1.

Next, the value of the variable val1 is given by the following (Equation 3a) (step 212b). val1 = 1 (but _{x 1 ≧ N-x 1)} val1 = 0 ( provided that _{x 1 <N-x 1)} ···· ( Formula 3a)

Further, the minimum value of the distance between the image of the image group B and the registered image is set to R ′ _min (step 212).
c). Then, the value of the variable val2 is given by the following (Equation 3b) (step 212d). val2 = R ′ _min / (R _min + R ′ _min ) (Equation 3b) Note that the flow of steps 212a to 212d is represented by process A in FIG. 3, and a specific flowchart is shown in FIG. FIG.

The relationship strength value between the registered image and the keyword K is selected by (Expression 7) (Step 213). Returning to step 204, it is determined whether calculation has been performed for all keywords.

[0049]

(Equation 7)

As described above, for all registered images,
Since all keyword-feature-value relationship strength values can be calculated, a keyword having a large keyword-feature-value relationship strength value can be automatically given as a keyword candidate for the image. In addition, since the keyword / feature amount relationship strength values of all the keywords are assigned, a result is obtained that the keyword / feature amount relationship strength values are similar if the keywords are similar. Therefore, it is possible to search for similar keywords without defining the similarity between keywords.

FIG. 4 is a flow chart showing a keyword search and a search result narrowing operation. First, a keyword for performing a keyword search is input using the keyword input unit 132 (step 301). Keyword search unit 131
Then, an image in which the keyword / feature relationship strength value of the input keyword is equal to or larger than the threshold α is searched from the registration database 104 (step 302), and the search result is displayed on the search result display unit 135 (step 303). If all the images in the search result cannot be displayed at once, only the number of images that can be displayed is displayed, and the images to be displayed are switched according to a user's instruction. From the displayed images, the image specifying unit 13
4 to specify an image to be narrowed down (step 30)
4). The displayed images of the search result are classified into an image group A designated in step 304 and an image group B not designated (step 305). Feature weight Ws
(t) is initialized (step 306), and it is determined whether the end condition of the feature amount weight calculation is satisfied (step 3).
07). If the termination condition is satisfied, step 30
The image feature amount of the image searched in step 2 is weighted by a feature amount weight Ws (t) (step 314), and is classified by a self-organizing map (hereinafter, SOM method) (step 315). The images classified into the same category as the image specified in step 304 are displayed as the narrowed search results (step 316). Also,
As the next candidate of the narrowed search result, a classification close to the classification displayed in step 316 is selected. If the termination condition is not satisfied in step 307, first, two images are randomly selected from the image group A classified in step 305 (step 308). The difference square vector d1 (t) of the image selected in step 308 is calculated (step 309). d1 (t) is selected by (Equation 1). One image is randomly selected from the image group A and one from the image group B classified in step 305 (step 310). Step 310
The difference square vector d2 (t) of the image selected in is calculated in the same manner as d1 (t) (step 311). The feature weight vector WK (t
+1) is set as (Equation 2) (step 312). Next, the process returns to step 307 with t = t + 1 (step 313).

Here, the network configuration of the SOM method is shown in FIG. 6 and will be described below. The network is composed of two layers, an input layer and a competitive layer (output layer). The number of units (input units) constituting the input layer is equal to the number of input signal vectors. All numbers are combined. On the other hand, the units (output units) constituting the competitive layer (output layer) are regularly arranged on a two-dimensional lattice.

Next, the classification operation according to the SOM method will be described with reference to the flowchart of FIG. The input to the input unit j at time t is Xj (t), and the weight of the link from the input unit j to the output unit k is Wkj (t) as shown in FIG. Further, as shown in FIG. 6A, the neighborhood set at the time t of the output unit k is N (k, t), and N (k, t1) for any time t1, t2 that satisfies t1 <t2. The area in the vicinity is reduced over time so that ⊂N (k, t2) holds. At this time, the weight Wkj of all links is set to a small random value (step 601).

An initial time is set to t = 0, and N (k, 0) is defined for all output units k. For example, let a large hexagon centered on k be N (k, 0). Here, the learning rate α (0) is determined as a function such that 0 <α (0) <1 and decreases with time, for example, a function such as Equation 8 (Step 602). .

(Equation 8)

If there is no t + 1-th data, the operation is stopped; otherwise, the t + 1-th data is input (step 603). Thereafter, the sum of the squares of Xj (t) -Wkj (t) for all output units k is calculated by (Equation 9) (step 60).
4).

(Equation 9)

For all the output units k, the minimum value of the value calculated in step 4 and the output unit k_min having the minimum value are obtained by (Equation 10) (step 60).
5).

(Equation 10)

All output units k∈N (k_min,
t), for all input units j,
1) (Step 606).

[Equation 11]

Thereafter, t = t + 1 is set, and step 60 is executed.
Go to 3.

As described above, steps 603 to 6
By repeating the operation up to 06, connection weights representing the distribution of the input vectors are formed for some output units.

FIG. 5 is a flowchart showing the operation of recalculating the feature amount weight for each keyword. A keyword similar to the keyword input in step 301 is read (step 40).
1). An image feature similar to the image specified in step 304 is read (step 402). One of the keywords read in step 401 is selected (step 403). The selected keyword is set as a keyword K.
A sample image similar to the sample image selected in step 102 or step 207 is selected (step 40).
4). The image read in step 402 is added to the sample image (step 405). The sample images are classified into an image group A with the keyword K and an image group B without the keyword K (step 40).
6). The image read in step 402 is the keyword K
Treated as an image with. Hereinafter, using image group A and image group B, feature amount weight WK of keyword K
Recalculate (t). The initial time is t = 0 and the keyword K
The feature weight vector WK (t) is initialized (step 4).
07). As the initial value, a value of a conventional weight vector or a random value is used. It is determined whether or not the end condition is satisfied (step 408). If the end condition is satisfied, it is determined whether or not the feature weights of all the keywords read in step 401 have been updated (step 41).
5) If all the feature amount weights have been updated, the process ends. If there is a keyword whose feature amount weight has not been updated, the process returns to step 403. If the end condition is not satisfied in step 408, two images are randomly selected from image group A (step 409). Difference square vector of the feature amount of the image selected in step 409
d1 (t) is selected by (Equation 1) (step 410).

One image is selected at random from the image group A and one from the image group B (step 411). The difference square vector d2 (t) of the two images is calculated in the same manner as d1 (t) (step 412). The feature amount weight vector WK (t + 1) is set as (Equation 2) (step 413). Next, the process returns to step 408 as t = t + 1 (step 414).

The recalculation of the keyword / feature-value relation strength value performs the same processing as the calculation of the keyword / feature-value relationship strength value by using the updated keyword-based feature value weight.
The images to be calculated are all the images registered in the registered image database.

It should be noted that the feature quantity can be implemented with feature quantities other than those shown in Table 1. It should be noted that the method of weighting the feature amount can be implemented by a method other than the method shown in Expression 2.

Furthermore, in the above description, an example of an image search system has been described as a data search system. However, the present invention is not limited to the search for image data, and is similarly applied to the search for other types of data. Is what you can do.

(Embodiment 2) Next, a second embodiment of the present invention will be described. In the first embodiment, the keywords are rearranged in descending order of the keyword / feature amount relationship strength value, and the top N keywords are assigned to each image.

On the other hand, in the present embodiment, the strength of the keyword / feature relation is regarded as the weight of the keyword in the registered image, and the K-NN method (K-nearest) is used.
Neighbor method) is applied. By applying the K-NN method, the distance between the registered image and the learning image is measured in the image feature space, and the top K images with the closest distance are selected. The keywords assigned to the selected K images are checked, and the N most frequently used keywords are assigned to the images. In this case, if the usage frequencies are equal, the keyword with the shortest distance is ranked higher.

Feature weights are calculated for all keywords. In the image feature space weighted by the feature amount weight, K images having a short distance are selected. Since each keyword is selected by the feature amount weight, a total of N _all × K is selected. The number of all keywords is set to N _all . Then, using the feature amount weights of all the keywords, K neighboring images are selected. Then, the keywords assigned to the total N _all × K images are checked, and the N most frequently used keywords are assigned to the images.

It should be noted that K used in the present embodiment is
The -NN method is one of methods for determining which class unknown data belongs to. In summary, the distance between data (learning data) whose class is already known and unknown data is measured, and the top K data whose distance is the closest (this K corresponds to K in the K-NN method) ) Is selected. Next, the class to which the learning data belongs is checked, and the class to which the learning data belongs most is determined as the class of the unknown data.

Further, the logic is further advanced, and the product of the weight of the keyword and the frequency of use is treated as a weighted use frequency, and the N keywords obtained by applying the K-NN method are added to the image. Can also be adopted. As described above, the keyword / feature amount relationship strength is regarded as the weight of the keyword in the registered image, and by applying the K-NN method, even if the data is unknown, the keyword easily belongs using the learning data ( Alternatively, a class that is most suitable to belong to) can be determined, and a keyword can be assigned to an image.

[0070]

As described above, the image retrieval system of the present invention automatically extracts a sensitivity word from an image in an image retrieval system using a sensitivity word as a keyword.
By reducing the load of image registration and combining search by combination of sensibility words and search refinement to directly specify an image, a user's intuitive search can be realized, and information generated in the process of search refinement operation can be reduced. By updating the learning so as to match the user's sensibility, it is possible to obtain an effect that an image search that can be adapted to the user's sensibility without causing a burden on the user can be performed.

Further, since the keyword-feature-value relationship strength values of all the keywords can be given, the result that the keyword-feature-value relationship strength values are similar for similar keywords is obtained. Therefore, an effect is obtained that a similar keyword can be searched without defining the similarity between keywords.

[Brief description of the drawings]

FIG. 1 is a block diagram of an image search system according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating an operation of calculating a feature amount weight for each keyword;

FIG. 3 is a flowchart showing an operation of calculating a keyword / feature amount relationship strength value;

FIG. 4 is a flowchart showing a keyword search and a search result narrowing operation;

FIG. 5 is a flowchart showing an operation of recalculating feature amount weights for each keyword;

FIG. 6 is a network configuration diagram of the SOM method.

FIG. 7 is a flowchart for explaining the operation of the SOM method;

FIG. 8 shows steps 212a to 21 in the flowchart of FIG. 3 showing the operation of calculating the keyword / feature amount relationship strength value.
Flow chart for explaining the details of 2d

FIG. 9 is a block diagram for explaining the principle of a conventional image search system.

FIG. 10 is a flowchart for explaining the principle of a conventional image search system.

FIG. 11 is a configuration diagram of a neural network for explaining the principle of a conventional image search system.

[Explanation of symbols]

 101 Keyword-based feature amount learning block 102 Registered image evaluation block 103 Search block 104 Registered image database 111 Sample image 112 Image feature amount extraction unit 113 Keyword-based feature amount weight calculation unit 114 Keyword-based feature amount weight 121 Registered image data 122 images Feature amount extraction unit 123 Keyword / feature amount relationship strength calculation unit 124 Keyword / feature amount relationship strength value 131 Keyword search unit 132 Keyword input unit 133 Search result narrowing down unit 134 Image designation unit 135 Search result display unit

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B050 EA04 FA02 FA12 FA19 GA08 5B075 ND06 NK02 NK07 NK08 PP28 PR06 QM08 QS03 QT04 5L096 BA08 EA02 EA05 EA07 EA12 EA39 FA54 FA68 FA81 GA41 GA59

Claims (12)

[Claims] 1. A data retrieval system for retrieving data by a keyword, a keyword input means for inputting a keyword, a data input means for inputting data,
A data storage unit for storing data feature amounts and keywords together with registration data; a keyword search unit for searching data by combining the keywords input from the keyword input unit; and automatically extracting keywords and feature amounts from the registration data A data search system comprising: a feature / keyword extraction unit; and a search result narrowing unit that narrows down a search result obtained by the keyword search unit without using a keyword. 2. An image search system for searching for an image by a keyword, a keyword input unit for inputting a keyword, an image input unit for inputting an image, an image storage unit for storing an image feature and a keyword together with a registered image, A keyword search unit for searching for an image by combining the keywords input from the keyword input unit, a feature amount / keyword extraction unit for automatically extracting a keyword or a feature amount from a registered image, and a result searched by the keyword search unit A search result narrowing means for narrowing down without using a keyword. 3. The search result narrowing means includes a display means for displaying a part or all of the images of the search result, and an image for designating an image similar to a desired image from the images displayed by the search result display means. Designating means, learning means for learning the characteristics of the desired image using the image data displayed by the search result display means as learning data, classification in accordance with the characteristics of the desired image, and output of the classification result to the display means 3. The image search system according to claim 2, further comprising a classification unit. 4. A feature / keyword extracting means for calculating the strength of the relationship between the keyword and the feature using the image data and the keyword registered in the image storage means; A keyword / feature that recalculates the strength of the relationship between a keyword and a feature using part or all of the search results narrowed down by the result narrowing means, the keyword specified in the keyword search, and the image data registered in the image storage unit. Quantity relation strength recalculation means, image selection means for automatically selecting an image used for calculation by the keyword / feature quantity relation strength calculation means from images stored in the image storage means, and an image for reading an image from the image storage means A reading unit, an image feature amount calculating unit that calculates a feature amount of the image read by the image reading unit, and an image feature amount calculating unit. The calculated feature amount and the keyword
Calculation result writing means for outputting to the image storage means the strength of the relationship between the keyword and the feature quantity calculated by the feature quantity relationship strength calculation means and the keyword / feature quantity relationship strength recalculation means. The image retrieval system according to claim 2 or 3, wherein 5. The image retrieval system according to claim 2, wherein the image storage means stores a value indicating the strength of the relationship between the image and all the keywords, separately from the keywords. . 6. A registered image database for storing a registered image, a sample image selected and read from a registered image database, an image feature extracting means for extracting an image feature of the sample image, and a keyword for the sample image. Keyword input means for adding or correcting already added keywords, and a keyword weighting feature that increases similarity between images with the same keyword and decreases similarity with images without the same keyword A feature amount weight learning device comprising: a keyword-based feature amount weight calculation unit that separately calculates, and obtains a feature amount weight for each keyword in a registered image. 7. A registered image database for storing registered images, an image feature extracting means for extracting image features from registered image data, and a keyword / feature for calculating a relation strength value between a keyword and a registered image feature. A registered image evaluation device, comprising: a relation strength calculating unit, for calculating a relation strength between a keyword and a registered image feature amount for a registered image. 8. A registered image database for storing a registered image, a sample image selected and read from a registered image database, an image feature extracting means for extracting an image feature of the sample image, and a keyword for the sample image. Keyword input means for adding or correcting already added keywords, and a keyword weighting feature that increases similarity between images with the same keyword and decreases similarity with images without the same keyword A feature amount weight learning unit including a keyword-based feature amount weight calculating unit separately calculated; a registered image database storing a registered image; an image feature amount extracting unit extracting an image feature amount from the registered image data; Keyword / feature relation strength calculation means for calculating the relation strength value between the image and the registered image feature And a registered image evaluation unit for calculating the relation strength between the keyword and the registered image feature amount for the registered image. The feature amount weight data output from the feature amount weight learning unit is used for the keyword / feature amount relationship of the registered image evaluation unit. A keyword setting device configured to set and update a keyword with respect to a registered image by performing an arithmetic process by inputting to a strength calculating means. 9. A registered image database for storing a registered image, a sample image selected and read from a registered image database, an image feature extracting means for extracting an image feature of the sample image, and a keyword for the sample image. Keyword input means for adding or correcting already added keywords, and a keyword weighting feature that increases similarity between images with the same keyword and decreases similarity with images without the same keyword A feature amount weight learning unit including a keyword-based feature amount weight calculating unit separately calculated; a registered image database storing a registered image; an image feature amount extracting unit extracting an image feature amount from the registered image data; Keyword / feature relation strength calculation means for calculating the relation strength value between the image and the registered image feature The registered image evaluation means for calculating the strength of the relationship between the keyword and the registered image feature amount for the registered image, and the feature amount weight data output from the feature amount weight learning means to the keyword / feature amount relationship of the registered image evaluation means. A keyword setting device for setting and updating a registered image by inputting to the intensity calculating means and performing an arithmetic processing, a keyword input means for performing a keyword search, and corresponding to a keyword from the registered images Keyword search means for searching registered images to be searched, image designating means for designating a part of images to be narrowed down from search results of the keyword search means, search result narrowing section for further narrowing down search results of the keyword search section, and search Image with keyword setting function, comprising: an image search device having a search result display unit for displaying results Search system. 10. The keyword / feature amount relationship strength calculation means performs a process of rearranging keywords in descending order of keyword / feature amount relationship strength values and assigning the top N keywords to each image. 9. The keyword setting device according to claim 8, wherein 11. The keyword / feature relation strength calculating means uses the keyword / feature relation strength as the weight of a keyword in a registered image, and measures the distance between the registered image and the learning image in the image feature space by the K-NN method. And
It is characterized in that a process of selecting the top K keywords of the images having a short distance, checking the keywords assigned to the selected K images, and giving the top N keywords most frequently used to the images is performed. The keyword setting device according to claim 8. 12. A product of a keyword weight and a use frequency is treated as a weighted use frequency, a distance between a registered image and a learning image is measured in an image feature space by a K-NN method, and the top K images of a close distance image 9. The keyword setting device according to claim 8, further comprising: selecting a keyword assigned to the selected K images, and performing a process of assigning the N most frequently used keywords to the image. .