JP6553664B2 - Model learning device, score calculation device, method, data structure, and program - Google Patents

Description

  The present invention relates to a model learning device, a score calculation device, a method, a data structure, and a program, and in particular, a model learning device, a score calculation device, a method, and data for calculating a similarity score between query data and reference data. It relates to structure and program.

  At the time of trademark application, similarity with registered trademark images is one of the criteria for registration examination.

  Many trademark images are described by abstracting a certain concept. For example, “person” may be represented by a combination of primitive symbols such as circles and squares, such as stick figures, while there are illustrations depicting the whole body in detail.

  It is very difficult to determine the similarity between the application trademark image and such a large number of abstract trademark images registered in the past from the viewpoint of time efficiency and objectivity.

  Under such circumstances, there is a demand for a method that supports the similarity determination of the abstracted concept by the trademark examiner (the user who conducts the preliminary search of the and / or application).

  Also, based on a set of queries, correct images, and incorrect images, the purpose is to learn so that the score between query-similar images is high and the score between query-non-similar images is low. Loss function is proposed (Non-Patent Document 1).

Jiang Wang et al., "Learning Fine-grained Image Similarity with Deep Ranking", Internet Search <URL: https://arxiv.org/pdf/1404.4661v1.pdf>

  It is an object of the present invention to provide a model learning device, a score calculation device, a method, a data structure, and a program that can calculate a similarity score with high accuracy.

  In order to achieve the above object, a model learning device according to a first aspect of the present invention is a model learning device that learns a multi-layered neural network for receiving data and outputting feature vectors, which includes query data and query data And the similar to the feature vector of the query data output from the multi-layered structure neural network that performs normalization on the output of the entire combined layer based on similar data similar to L and dissimilar data not similar to the query data. The feature vector of the query data and the feature of the similar data using a loss function including the similarity score of the data to the feature vector and the similarity score of the feature vector of the query data and the feature vector of the dissimilar data The similarity score with the vector is high, and the feature vector of the query data and the As the similarity score between the feature vector data it is lowered, and is configured to include a learning unit for learning the parameters of the multi-layered structure neural network.

  A model learning method according to a second aspect of the present invention is a model learning method in a model learning device for learning a multi-layered structure neural network for inputting data and outputting a feature vector, wherein the learning unit comprises query data and query data And the similar to the feature vector of the query data output from the multi-layered structure neural network that performs normalization on the output of the entire combined layer based on similar data similar to L and dissimilar data not similar to the query data. Using a loss function including a similarity score with a feature vector of data and a similarity score between the feature vector of the query data and the feature vector of the dissimilar data, the feature vector of the query data and the feature of the similar data The similarity score with the vector is high, and the feature vector of the query data As the similarity score between the feature vector of the dissimilar data is low, it learns the parameters of the multi-layered structure neural network.

  A model learning device according to a third aspect of the present invention is a model learning device for learning a multi-layered structure neural network for inputting data and outputting a feature vector, comprising: query data; similar data similar to the query data; Similarity score between the feature vector of the query data output from the multilayer structure neural network and the feature vector of the similar data based on dissimilar data not similar to the query data, the dissimilarity of the query data and the feature vector Similarity score between data and feature vector, similarity score between the query data and the similar data calculated using information different from the feature vector, and calculated using information different from the feature vector Loss including similarity score between the query data and the dissimilar data The similarity score between the feature vector of the query data and the feature vector of the similar data is high, and the similarity score between the feature vector of the query data and the feature vector of the dissimilar data is low, using a number. And a learning unit configured to learn parameters of the multi-layered neural network.

  A model learning method according to a fourth aspect of the invention is a model learning method in a model learning device for learning a multi-layered structure neural network for inputting data and outputting a feature vector, the learning unit comprising query data and query data A similarity score between a feature vector of the query data output from the multi-layer structure neural network and a feature vector of the similar data based on similar data similar to the data and dissimilar data not similar to the query data; The similarity score between the feature vector of data and the feature vector of the dissimilar data, the similarity score between the query data and the similar data calculated using information different from the feature vector, and the feature vector The dissimilarity to the query data calculated using different information Similarity score between the feature vector of the query data and the feature vector of the similar data is high using a loss function including the similarity score with the data, and the feature vector of the query data and the feature vector of the dissimilar data The parameters of the multi-layered neural network are trained so that the similarity score with.

  A score calculation device according to a fifth aspect of the present invention is a score calculation device for calculating a score representing the degree of similarity between query data and reference data, wherein all joint layers learned in advance based on query data and reference data Calculating a feature vector of the query data output from the multilayer structure neural network that performs normalization on the output of the second output data and a feature vector of the reference data output from the multilayer structure neural network, the feature vector of the query data A score calculation unit for calculating a similarity score with the feature vector of the reference data, and another method score for calculating a similarity score between the query data and the reference data using information different from the feature vector A calculation unit, the similarity score calculated by the score calculation unit, and the other method score calculation unit Thus it is configured to include a score integration unit that calculates a calculated combined score that integrates with the similarity score, the.

  A data structure according to a sixth aspect of the present invention is a parameter used to calculate a similarity score between query data and reference data, the input comprising query data, similar data similar to the query data, The non-similar data not similar to the query data, and the output being the normalized feature vector of the query data, the normalized feature vector of the similar data, and the non-similar data normalization to obtain cosine similarity It is a data structure including the parameters of the multi-layered structure neural network as a set feature vector.

  A program according to a seventh aspect is a program for causing a computer to function as each part constituting the model learning device or the score calculation device.

  As described above, according to the model learning apparatus, method, and program of the present invention, the similarity score between the feature vector of the query data and the feature vector of the similar data is high using the loss function, and the query By learning the parameters of a multilayered neural network that normalizes the output of all connected layers so that the similarity score between the feature vector of the data and the feature vector of the dissimilar data is low, the similarity is accurately obtained. A model for calculating the degree score can be learned.

  According to the score calculation apparatus and program of the present invention, the feature vector of the query data output from the multilayer structure neural network that performs normalization on the output of all the combined layers, and the reference data output from the multilayer structure neural network The feature score is calculated and the similarity score is calculated. Using the information different from the feature vector, the similarity score between the query data and the reference data is calculated and integrated to calculate the similarity score with high accuracy. can do.

  According to the data structure of the present invention, the input is query data, similar data, non-similar data, and the output is normalized feature vector of query data for finding cosine similarity, normalized similar data A model for calculating the similarity score with high accuracy can be learned by using the parameters of the multilayered neural network as the feature vector and the normalized feature vector of the dissimilar data.

(A) The figure which shows the structure of DNN "Illustration2Vec", and (B) The figure which shows the structure of the DNN model of the 1st Embodiment of this invention. It is a figure which shows the example which represented each brand image in binary with the presence or absence of each figure as a bit. FIG. 1 is a block diagram showing a configuration of a model learning device according to a first embodiment of the present invention. It is a block diagram showing composition of a score calculation device concerning a 1st embodiment of the present invention. It is a flowchart which shows the content of the model learning process routine in the model learning apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the content of the score calculation process routine in the score meter apparatus which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the method to integrate the score obtained by using a DNN feature vector, and the score obtained by the other method. It is a figure for demonstrating the method of learning the parameter of DNN using the loss function containing the score obtained using the DNN feature vector, and the score obtained by the other method. It is a figure for demonstrating the method to integrate the score obtained by using a DNN feature vector, and the score obtained by the other method. It is a block diagram which shows the structure of the model learning apparatus which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the score calculation apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the content of the model learning process routine in the model learning apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the content of the score calculation process routine in the score scale apparatus which concerns on the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Overview of the embodiment of the present invention>
The DNN for illustration image classification “Illustration2Vec” is designed for the purpose of “multi-class classification of general illustrations” (see FIG. 1A). Therefore, the DNN according to the embodiment of the present invention has the following three points. It is improving (refer to FIG. 1 (B)).

(1) Exclude color information (use grayscale to make input image one channel).
(2) Introduce a normalization layer to adapt to cosine similarity.
(3) A loss function Triplet-loss suitable for search ranking is introduced.

  Further, the normalization layer performs a process (L2 normalization) of dividing each element by a value obtained by squaring and summing each element of the vector.

... (1)

  Here, y is the output value of the normalization layer, and corresponds to a feature vector. Further, x is the output value of the previous layer (in the example of FIG. 1B above, a 4096-dimensional vector that is the output of all coupling layers). Calculate Triplet Loss using y.

  Here, the reason for introducing the normalization layer will be described.

  Distance calculation of Triplet Loss is performed by Euclidean distance. On the other hand, in general, cosine similarity is often suitable for the distance (similarity) in image search.

  The Euclidean distance d is expressed by the following equation.

  The cosine similarity cos is expressed by the following equation.

  When L2 normalization is performed, || x || = 1, so the Euclidean distance and the cosine similarity are equivalent. Therefore, it is possible to adapt to the cosine similarity by introducing a normalization layer and performing L2 normalization on the output values of all coupling layers.

  Next, a method for selecting Triplet data will be described.

  Although the calculation of TripletLoss requires correct data consisting of both similar and dissimilar images for a given query image, similarity in search is often determined by subjective evaluation and also for DNN learning. The cost of data collection is high because a large amount of data is required.

  In the embodiment of the present invention, information other than the image is used to select the Triplet "non-similar image".

  For example, all registered trademark images are classified based on Vienna figures. The graphic code of the query image is compared with the graphic code of any trademark image, and the number of data of the non-similar image is increased by making the non-similar image a trademark image having a low degree of code coincidence.

  For example, as shown in FIG. 2, each trademark image is represented in binary with the presence or absence of each figure as a bit, the matching degree is calculated by the Hamming distance, and the trademark image having the Hamming distance equal to or more than the threshold is regarded as a dissimilar image.

First Embodiment
<System Configuration of Model Learning Device>
FIG. 3 is a block diagram showing the model learning apparatus 100 according to the first embodiment of the present invention. The model learning apparatus 100 is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a model learning processing routine described later, and is functionally configured as follows. ing.

  As shown in FIG. 3, the model learning device 100 according to the present embodiment includes an input unit 10, a calculation unit 20, and an output unit 40.

  The input unit 10 receives, as learning data, a plurality of pairs of a query image and a similar image similar to the query image.

  The calculation unit 20 includes a learning data storage unit 21, an image database 22, a dissimilar data selection unit 23, and a learning unit 24.

  The learning data storage unit 21 stores a plurality of pairs of a query image received by the input unit 10 and a similar image similar to the query image as learning data.

  The image database 22 stores a set of reference images.

  The dissimilar data selection unit 23 uses, for each of the query images of the learning data, information other than the image, and from the set of reference images stored in the image database 22, dissimilar images that are not similar to the query image. Choose

  The learning unit 24 learns DNN parameters based on a plurality of pairs of query images and similar images of learning data, and dissimilar images selected for each of the query images. The DNN according to the present embodiment performs normalization on the outputs of all the combined layers and outputs a feature vector.

  In learning the parameters of DNN, the query image is input and the feature vector (output of the normalization layer shown in the above equation (1)) input from the DNN and the similar image is input and the feature vector output from DNN ( The similarity score with the output of the normalized layer shown in the above equation (1), the feature vector of the query image (the output of the normalized layer shown in the above equation (1)) and the dissimilar image are input and output from DNN The similarity score between the feature vector of the query image and the feature vector of the similar image is high, using the loss function including the similarity score with the feature vector (the output of the normalized layer shown in the above equation (1)) The parameters of DNN are learned so that the similarity score between the feature vector of the query image and the feature vector of the dissimilar image is low. The DNN according to the present embodiment includes, as shown in FIG. 1 (B), an input layer to which one channel of an input image is input, a convolutional layer, a total coupling layer, and a normalization device. It consists of

  A conventionally known method (for example, error back propagation of general Deep Learning) may be used as a learning method.

  Note that the DNN parameters may be learned by using (part of) existing appropriate learning parameters as initial values of the DNN parameters. For example, learning parameters by Illustration 2 Vec or VGG may be used. This increases the learning effect.

  The learned parameters of DNN are output by the output unit 40. Here, the parameters of DNN are parameters used to calculate the similarity score between the query image and the reference image, and the input is a query image, a similar image, a dissimilar image, and the output is a cosine Data structure including parameters of multi-layer structure neural network which is normalized feature vector of query image, normalized feature vector of similar image, and normalized feature vector of dissimilar image for finding similarity is there.

<System Configuration of Score Calculation Device>
FIG. 4 is a block diagram showing the score calculation apparatus 150 according to the first embodiment of the present invention. The score calculation device 150 is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a score calculation processing routine described later, and is functionally configured as follows. ing.

  As shown in FIG. 4, the score calculation device 150 according to the present embodiment includes an input unit 60, a calculation unit 70, and an output unit 90.

  The input unit 60 receives an input of a query image.

  The calculation unit 70 includes a model storage unit 71, an image database 72, and a score calculation unit 73.

  The model storage unit 71 stores parameters of DNN output by the model learning device 100.

  The image database 72 stores a set of reference images.

  The score calculation unit 73 receives the query image based on the set of the query image and the reference image, and outputs the feature vector (the output of the normalized layer shown in the above equation (1)) input from the DNN and the reference image. To obtain a feature vector (output of the normalized layer shown in the above equation (1)) output from DNN, and for each of the reference images, a feature vector of the query image and a feature vector of the reference image A similarity score representing the cosine similarity is calculated.

  The output unit 90 outputs the reference image in descending order of the similarity score.

<Operation of Model Learning Device>
Next, the operation of the model learning device 100 according to the first embodiment will be described. When a plurality of pairs of a query image and a similar image similar to the query image are input to the model learning device 100 as learning data, the model learning processing routine shown in FIG.

  First, in step S100, for each of query images of learning data, a non-similar image not similar to the query image is selected from a set of reference images stored in the image database 22 using information other than the image. Do.

  Then, in step S102, a query image is input based on a plurality of pairs of query images and similar images of learning data and non-similar images selected for each of the query images, and a feature vector output from DNN Loss function including similarity score with feature vector input from similar image and output from DNN, and similarity score with feature vector from query image and non-similar image output from DNN Learn the parameters of DNN to optimize. The learned parameters of DNN are output by the output unit 40, and the model learning processing routine is ended.

<Operation of score calculation device>
Next, the operation of the score calculation apparatus 150 according to the first embodiment will be described. First, the DNN parameters output from the model learning device 100 are input to the score calculation device 150 and stored in the model storage unit 71. Then, when the query image is input to the score calculation device 150, the score calculation processing routine shown in FIG.

  First, in step S110, a query image is input to obtain a feature vector output from DNN.

  In step S112, a reference image stored in the image database 72 is input to obtain a feature vector output from the DNN.

  In step S114, a similarity score representing the cosine similarity between the feature vector of the query image obtained in step S110 and the feature vector of the reference image obtained in step S112 is calculated.

  In step S116, it is determined whether or not the processing of steps S112 to S114 has been performed for all reference images stored in the image database 72, and the reference images for which the processing of steps S112 to S114 have not been executed are If it exists, the process returns to step S112, and a feature vector of the reference image is obtained. On the other hand, when the processes in steps S112 to S114 have been executed for all reference images stored in the image database 72, the process proceeds to step S118.

  In step S118, reference images are output by the output unit 90 in descending order of the similarity score calculated in step S114, and the score calculation processing routine ends.

  As described above, according to the model learning device according to the first embodiment of the present invention, the similarity score between the feature vector of the query image and the feature vector of the similar image is high using the loss function, and the query Calculate the similarity score with high accuracy by learning the DNN parameter that normalizes the output of all connected layers so that the similarity score between the feature vector of the image and the feature vector of the dissimilar image is low To learn a model to do.

  Further, according to the score calculation device according to the first embodiment of the present invention, the similarity score between the feature vector of the query image and the feature vector of the similar image based on the parameters of DNN learned by the model learning device. The similarity score can be calculated with high accuracy by calculating

Second Embodiment
Next, a second embodiment will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  In the second embodiment, the DNN is integrated with the score obtained by another method different from the DNN to calculate the similarity score, and using the loss function including the score obtained by the other method, the DNN The point of learning the parameters is different from that of the first embodiment.

<Overview of this embodiment>
As shown in FIG. 7, in the present embodiment, a score using a feature vector output by DNN and a score obtained by another method are integrated to obtain a final score. Specifically, the score (cosine similarity) y using the feature vector output by the DNN described above and the score x obtained by another method are integrated by a weighted linear sum ax + by of both scores, and the final result is obtained. Ask for a score.

  Here, a score calculated by any method can be used as the score obtained by another method. For example, the degree of tag relevance attached to the image, the degree of coincidence of the Vienna graphic code, etc. can be used.

  The score integration can be replaced by any method. For example, the scores may be integrated by an expression similar to the expression incorporated in the DNN loss function, or the scores may be integrated by combining with SVM or Rank-SVM.

  Further, as shown in FIG. 8, DNN parameters are learned using a loss function including a score using a feature vector output by DNN and a score obtained by another method.

  The loss function in the present embodiment is expressed by the following equation (2).

... (2)

  However, f () represents a DNN feature vector, and g () represents a feature vector according to another method. Here, the first two terms relate to the degree of similarity of DNN feature vectors, and it is possible to learn only differences taking into account the effects of other methods, thanks to the later fixed terms (third term).

  The third term is the similarity (fixed value) between the query image and the similar image by another method, and the fourth term is the similarity (fixed value) between the query image and the dissimilar image by the other method The fifth term represents a margin (fixed value). When the value after the third term is negative, the value after the third term is set to 0.

  In addition, the score by another method may be plural, and it can respond by adding the same three terms as the latter three terms.

  Margin m is a parameter used only at the time of learning, and is tuned in accordance with the degree of similarity calculated by another method. For example, if the similarity based on the DNN feature vector is in the range of 0.00 to 1.00 and the similarity in the range of other methods is 100.0 to 101.00, m = -100 or the like.

  Learn the parameters of DNN by back-propagating to the parameters of DNN so as to optimize the above loss function.

  Here, three items and four items are written in the form of distance, but the contents of g do not matter, and the values of “similarity between query image and similar image” and “similarity between query image and dissimilar image” If it is required, other forms may be used.

  The similarity score using the feature vector output by DNN learned as described above and the similarity score obtained by another method are integrated to obtain an integrated score with the reference image, and a reference similar to the query image Search for images For example, as shown in the following formula, a simple sum is an integrated score (see FIG. 9).

<System Configuration of Model Learning Device>
FIG. 10 is a block diagram showing a model learning apparatus 200 according to the second embodiment of the present invention. The model learning apparatus 200 is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a model learning processing routine described later, and is functionally configured as follows. ing.

  As shown in FIG. 10, model learning apparatus 200 according to the present embodiment includes input unit 10, calculation unit 220, and output unit 40.

  The input unit 10 receives, as learning data, a plurality of pairs of a query image and a similar image similar to the query image.

  The calculation unit 220 includes a learning data storage unit 21, an image database 22, a dissimilar data selection unit 23, another method score calculation unit 223, a learning unit 224, a score calculation unit 226, and a weight learning unit 228.

  The other method score calculation unit 223 generates, for each of the query images of the learning data, a similarity score with a similar image of the query image and a similarity score with a dissimilar image that is not similar to the query image. It is calculated by another method different from the vector.

  The learning unit 24 uses a plurality of pairs of query images and similar images of learning data, dissimilar images selected for each of the query images, and the similarity between the query image calculated by the other technique score calculation unit 223 and the similar images. The parameters of DNN are learned based on the score and the similarity score between the query image and the dissimilar image calculated by the other method score calculation unit 223.

  In DNN parameter learning, a feature vector (output of the normalization layer shown in the above equation (1)) input from a query image and output from the DNN, and a feature vector (output from the DNN input from a similar image) The similarity score with the output of the normalization layer shown in the above equation (1), the feature vector of the query image (the output of the normalization layer shown in the above equation (1)) and the dissimilar image are inputted and outputted from the DNN. The similarity score with the feature vector (the output of the normalization layer shown in the above formula (1)), the similarity score between the query image and the similar image calculated by the other method score calculation unit 223, and the other method score calculation unit The similarity score between the feature vector of the query image and the feature vector of the similar image is high using the loss function shown in the above formula (2) including the similarity score between the query image and the dissimilar image calculated by 223. As the similarity score between the feature vector of the feature vectors and the non-similar image of the query image is lowered, to learn the parameters of DNN.

  A conventionally known method (for example, error back propagation of general Deep Learning) may be used as a learning method.

  The learned parameters of DNN are output by the output unit 40.

  The score calculation unit 226 inputs a query image based on a plurality of pairs of query images and similar images and learned DNN parameters, and outputs a feature vector (normalization shown in the above formula (1)). Layer output) and a feature vector (output of the normalization layer shown in the above formula (1)) output from the DNN by inputting a similar image, and for each of a plurality of pairs of a query image and a similar image, A similarity score representing cosine similarity between the feature vector of the query image and the feature vector of the similar image is calculated.

  In addition, the score calculation unit 226 inputs a query image based on a plurality of pairs of query images and dissimilar images and learned DNN parameters, and outputs a feature vector (in the above equation (1)). Output of the normalization layer shown) and a feature vector (output of the normalization layer shown in the above formula (1)) inputted from the dissimilar image and outputted from the DNN, and a plurality of pairs of the query image and dissimilar image A similarity score representing a cosine similarity between the feature vector of the query image and the feature vector of the dissimilar image is calculated.

  Based on the similarity score y calculated by the score calculation unit 226 and the similarity score x calculated by the other technique score calculation unit 223, the weight learning unit 228 calculates an integrated score for the query image and similar image pair. The weights a and b for obtaining the integrated score are learned so that (ax + by) becomes high and the integrated score (ax + by) for the pair of the query image and the dissimilar image becomes low. The learned weight is output by the output unit 40.

<System Configuration of Score Calculation Device>
FIG. 11 is a block diagram showing a score calculation apparatus 250 according to the second embodiment of the present invention. The score calculation device 250 is composed of a computer including a CPU, a RAM, and a ROM that stores a program for executing a score calculation processing routine described later, and is functionally configured as follows. ing.

  The score calculation apparatus 250 which concerns on this Embodiment is provided with the input part 60, the calculating part 270, and the output part 90, as shown in FIG.

  The input unit 60 receives an input of a query image.

  The calculation unit 270 includes a model storage unit 71, an image database 72, a score calculation unit 73, another method score calculation unit 274, and a score integration unit 275.

  The model storage unit 71 stores DNN parameters output by the model learning device 200.

  The other method score calculation unit 274 uses a different method different from the feature vector output from the DNN based on the set of the query image and the reference image, for each reference image, the query image, the reference image, Calculate a similarity score that represents the similarity of

  The score integration unit 275 uses the weights output from the model learning device 200, and for each reference image, the similarity score y with the reference image calculated by the score calculation unit 73 and the other technique score calculation unit The similarity score x with the reference image calculated by 274 is integrated to calculate an integrated score (ax + by).

  The output unit 90 outputs the reference image in the descending order of the integrated score.

<Operation of Model Learning Device>
Next, the operation of the model learning device 200 according to the second embodiment will be described. When a plurality of pairs of a query image and a similar image similar to the query image are input as learning data to the model learning device 200, the model learning processing routine shown in FIG.

  First, in step S100, for each query image of learning data, a non-similar image that is not similar to the query image is selected from a set of reference images stored in the image database 22 using information other than the image. To do.

  In step S200, for each of the query images of the learning data, the similarity score with the similar image of the query image and the similarity with the dissimilar image not similar to the query image selected in step S100 The score is calculated by another method different from the DNN feature vector.

  In step S202, a plurality of pairs of query images and similar images of learning data, dissimilar images selected for each of the query images in step S100, and similarity between the query image and similar images calculated in step S200 Based on the score and the similarity score between the query image and the dissimilar image calculated in step S200, the query image is input and the feature vector output from the DNN, and the similar image is input and output from the DNN. The similarity score with the feature vector, the similarity score with the feature vector output from the DNN by inputting the feature vector and the dissimilar image of the query image, the similarity between the query image and the similar image calculated in step S200 above A degree score, and a similarity score between the query image and the dissimilar image calculated in step S200, So as to optimize the loss function shown in serial formula (2), to learn the parameters of DNN. The learned parameters of DNN are output by the output unit 40.

  Also, in step S204, based on the query image and a plurality of pairs of similar images and parameters of the learned DNN, the query image is input and the feature vector output from DNN and the similar image are input and the DNN is input. An output feature vector is obtained, and a similarity score representing a cosine similarity between the feature vector of the query image and the feature vector of the similar image is calculated for each of a plurality of pairs of the query image and the similar image.

  In addition, based on a plurality of pairs of a query image and a dissimilar image, and parameters of the learned DNN, the query image is input and a feature vector output from the DNN and a dissimilar image are input and output from the DNN. And a similarity score representing a cosine similarity between the feature vector of the query image and the feature vector of the dissimilar image is calculated for each of a plurality of pairs of the query image and the dissimilar image.

  In step S206, based on the similarity score calculated in step S204 and the similarity score calculated in step S200, the integrated score for the query image and similar image pair increases, and The weight for learning the integrated score is learned by weighted addition so that the integrated score for the query image and the dissimilar image pair is low. The learned weight is output by the output unit 40, and the model learning processing routine is ended.

<Operation of score calculation device>
Next, the operation of the score calculation apparatus 250 according to the second embodiment will be described. First, DNN parameters output from the model learning device 200 are input to the score calculation device 250 and stored in the model storage unit 71. Then, when the query image is input to the score calculation device 250, the score calculation processing routine shown in FIG.

  First, in step S110, a query image is input and a feature vector output from the DNN is obtained.

  In step S112, a reference image stored in the image database 72 is input to obtain a feature vector output from the DNN.

  In step S114, a similarity score representing the cosine similarity between the feature vector of the query image obtained in step S110 and the feature vector of the reference image obtained in step S112 is calculated.

  In step S210, the similarity between the query image and the reference image is determined for each reference image using another method different from the feature vector output from the DNN based on the set of query images and reference images. Calculate the similarity score to represent.

  In step S212, using the weights output by the model learning device 200, the similarity score with the reference image calculated by the score calculation unit 73 and the other method score calculation unit 274 are calculated for each reference image. The similarity score with the reference image is integrated by weighted addition to calculate an integrated score.

  In step S116, it is determined whether or not the processing in steps S112 to S114 and S210 to S212 has been performed for all reference images stored in the image database 72, and the processing in steps S112 to S114 and S210 to S212 is performed. If there is a reference image that has not been executed, the process returns to step S112, and a feature vector of the reference image is obtained. On the other hand, when the processes of steps S112 to S114 and S210 to S212 are executed for all reference images stored in the image database 72, the process proceeds to step S118.

  In step S212, the reference image is output by the output unit 90 in the descending order of the integrated score calculated in step S212, and the score calculation processing routine is ended.

  As described above, according to the model learning device according to the second embodiment of the present invention, the similarity score between the feature vector of the query image and the feature vector of the similar image, the feature vector of the query image and the dissimilar image Loss function including similarity score with feature vector, similarity score between query image and similar image calculated by other methods, and similarity score between query image and dissimilar image calculated by other methods The output of all connected layers is such that the similarity score between the feature vector of the query image and the feature vector of the similar image is high and the similarity score between the feature vector of the query image and the feature vector of the dissimilar image is low. By learning the parameters of DNN that performs normalization, it is possible to learn a model for calculating the similarity score with high accuracy. In particular, for an image that can be judged well with a DNN feature vector by backpropagating errors only to the DNN side, including the similarity score (fixed value) obtained by other methods calculated irrespective of DNN in the loss function. It is possible to learn the parameters of DNN so that the image can be judged similar by other methods well, as it is better.

  Moreover, according to the score calculation apparatus which concerns on the 2nd Embodiment of this invention, the feature vector of the query image output from DNN which normalizes with respect to the output of all the connection layers, and the reference output from DNN By calculating the image feature vector, calculating the similarity score, calculating the similarity score between the query image and the reference image by another method, and integrating them, the similarity score can be calculated with high accuracy.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the scope of the present invention.

  For example, although the above embodiment has been described by way of example in which DNN includes a convolutional layer and an all bonding layer, the present invention is not limited to this. For example, the part composed of the convolution layer and the total coupling layer can be replaced with an arbitrary network structure such as RNN or LSTM.

  Also, any network structure such as CNN, RNN, LSTM, etc. may be used as the multilayer structure neural network.

  Further, the case where the image similarity score is calculated has been described as an example. However, the present invention is not limited to this, and a moving image or audio similarity score may be calculated for a moving image or audio.

  The model learning devices 100 and 200 and the score calculation devices 150 and 250 described above have a computer system inside. However, if the “computer system” uses a WWW system, a homepage providing environment ( Or display environment).

  For example, although the present invention has been described as an embodiment in which the program is installed in advance, the program may be provided by being stored in a computer readable recording medium.

10, 60 Input unit 20, 70, 220, 270 Calculation unit 21 Learning data storage unit 22 Image database 23 Dissimilar data selection unit 24, 224 Learning unit 40 Output unit 70 Calculation unit 71 Model storage unit 72 Image database 73, 226 Score Calculation unit 90 Output unit 100, 200 Model learning device 150, 250 Score calculation device 223, 274 Other method score calculation unit 228 Weight learning unit 275 Score integration unit

Claims (8)

A model learning device for learning a multilayered neural network for inputting data and outputting a feature vector,
The query data output from the multilayer neural network that normalizes the output of the fully connected layer based on query data, similar data similar to the query data, and dissimilar data not similar to the query data A feature vector of the query data using a loss function including a similarity score between the feature vector of the query data and the feature vector of the similar data, and a similarity score between the feature vector of the query data and the feature vector of the dissimilar data And learning the parameters of the multilayer neural network so that the similarity score between the feature vector of the similar data and the feature vector of the query data is high and the similarity score between the feature vector of the query data and the feature vector of the dissimilar data is low Model learning device including a learning unit. A dissimilar data selection unit that selects the dissimilar data that is not similar to the query data from a set of reference data using information other than the data;
The model learning device according to claim 1, wherein the learning unit learns parameters of the multilayered neural network based on the query data, the similar data, and the selected dissimilar data. A model learning device for learning a multilayered neural network for inputting data and outputting a feature vector,
Based on query data, similar data similar to the query data, and dissimilar data not similar to the query data, a feature vector of the query data and a feature vector of the similar data output from the multilayer neural network A similarity score, a similarity score between a feature vector of the query data and a feature vector of the dissimilar data, a similarity score between the query data and the similar data calculated using information different from the feature vector, And using a loss function including a similarity score between the query data and the dissimilar data calculated using information different from the feature vector, the feature vector of the query data and the feature vector of the similar data The similarity score is high, and the feature vector of the query data and the dissimilarity As the similarity score between the feature vector of the motor is low, the model learning device including a learning section for learning the parameters of the multi-layered structure neural network. A score calculation device that calculates a score representing the similarity between query data and reference data,
Based on the query data and the reference data, the feature vector of the query data output from the multilayer neural network that is pre-trained and normalizes the output of the fully connected layer, and is output from the multilayer neural network. A score calculation unit that calculates a similarity score between the feature vector of the query data and the feature vector of the reference data;
Other method score calculation unit for calculating a similarity score between the query data and the reference data using information different from the feature vector;
A score integration unit that calculates an integrated score obtained by integrating the similarity score calculated by the score calculation unit and the similarity score calculated by the other method score calculation unit;
Score calculation device including. A parameter used to calculate a similarity score between query data and reference data,
The input is query data, similar data similar to the query data, dissimilar data not similar to the query data, and the output is a normalized feature vector of the query data for determining cosine similarity, the similarity A data structure including a normalized feature vector of data and a parameter of a multilayered neural network as a normalized feature vector of the dissimilar data. A model learning method in a model learning device for learning a multilayered neural network for inputting data and outputting a feature vector,
Based on query data, similar data similar to the query data, and dissimilar data not similar to the query data, the learning unit is output from the multilayer neural network that normalizes the output of the fully connected layer The query using a loss function including a similarity score between a feature vector of the query data and a feature vector of the similar data, and a similarity score between the feature vector of the query data and the feature vector of the dissimilar data The multilayer neural network has a high similarity score between the feature vector of the data and the feature vector of the similar data, and a low similarity score between the feature vector of the query data and the feature vector of the dissimilar data. Model learning method to learn parameters. A model learning method in a model learning device for learning a multilayered neural network for inputting data and outputting a feature vector,
Based on query data, similar data similar to the query data, and dissimilar data not similar to the query data, a learning unit outputs a feature vector of the query data and the similar data output from the multilayer neural network. A similarity score with a feature vector, a similarity score between a feature vector of the query data and a feature vector of the dissimilar data, and the query data calculated using information different from the feature vector and the similar data Using a loss function including a similarity score and a similarity score between the query data and the dissimilar data calculated using information different from the feature vector, the feature vector of the query data and the similarity data The similarity score with the feature vector is high, and the feature vector of the query data As the similarity score between the feature vector of the dissimilar data is lower, the model learning method for learning the parameters of the multi-layered structure neural network.   The program for functioning a computer as each part which comprises the model learning apparatus in any one of Claims 1-3, or the score calculation apparatus of Claim 4.