JP2006127446A - Image processing device, image processing method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing device capable of discriminating a color image or a gray scale image on which a character and an image or the like other than the character are mixed into a plurality of classes with high accuracy. <P>SOLUTION: This image processing device discriminates image information by a discriminator which learns on the basis of learning data, and comprises a feature amount extracting means for extracting a feature amount from the image information, a feature calculating means for calculating a feature which is a combination of the feature amounts extracted by the feature amount extracting means, a learning means for learning the discriminator by the feature amount calculated by the feature calculating means and the feature amount extracted by the feature amount extracting means, a collating means for collating the discrimination result, by applying teacher data to the discriminator having learned by the learning means, with an ideal discrimination result given from an external, and an optimizing means for changing a combination method of the feature amount by the feature calculating means on the basis of a collation result by the collating means. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus, an image processing method, a program, and a recording medium, and in particular, by determining a color or monochrome image in which characters and photographs are mixed into a plurality of classes, character determination, object determination, region determination, etc. Discrimination technology applicable to

In image processing, character / document image recognition processing has been conventionally performed. When performing this recognition processing, it is indispensable to obtain the correct position information of the character area occupied by the character / document image existing in a part of the processing target image in order to obtain high recognition accuracy.
For example, when character recognition processing is performed on a region other than a character region in an image, not only does it take time to perform unnecessary processing, but also as a result of forcibly performing character recognition on a region where no character exists, A lot of errors will be output.

  For this reason, in the technique described in Patent Document 1, an input image is reduced to obtain a circumscribed rectangle of a connected component of black pixels, and is classified into characters, tables, figures, and others based on the circumscribed rectangle. Then, the character elements are taken out of them and integrated to generate lines, and the generated lines are integrated to acquire the character area. Further, in this example, column information is extracted from the character region, and the character region that is excessively integrated is corrected with reference to the position of the extraction step.

On the other hand, with the recent spread of color printers and the like, character recognition based on color originals is increasing. When such a color original is used, the technique disclosed in Patent Document 1 cannot be applied unless the color image is converted into a binary image.
As a technique for solving this problem and extracting a character region from a color image in which characters and photographs are mixed, there are Patent Document 2, Non-Patent Document 1, and Non-Patent Document 2.

  In the technique described in Patent Document 2, a compressed image is generated from an original image, pixels that can be regarded as the same color are extracted as runs, their connected components are obtained for each color, and the obtained connected components are regarded as character candidates and are adjacent to each other. By combining connected components to generate a character line, and then extracting the character line extracted from the extracted character line, the character line can be output without the concept of background. It is what. As a result, by directly using the pixel information of the color image, it is possible to perform more accurate character region extraction, and to cope with even when the background color continuously changes.

In Non-Patent Document 1, character strings are extracted with high accuracy by relying on clusters in a color space based on prior knowledge that characters have the same color and size. Similarly, Non-Patent Document 2 also extracts character regions from color images such as magazine covers.
JP 2000-67158 A JP 2002-288589 A H. Kasuga, M. Okamoto and H. Yamamoto, `` Extraction of characters from color documents '', Proceedings of the SPIE-The International Society for Optical Engineering, V 3967, pp. 278-285, 2000. H.Hase, T.Shinokawa, M.Yoneda, CYSuen, `` Character string extraction from color documents '', Pattern Recognition 34, pp.1349-1365, 2001.

However, the technique described in Patent Document 2 does not have a concept of background, and a pixel block other than a character having an arrangement similar to a character as well as a character may be extracted overlapping the character even in the background. is there.
Further, Non-Patent Document 1 does not recognize which cluster is a character string, and the image being handled is a small one such as a poster card, so that the character and the background are clearly distinguished. It was an area discrimination for a relatively simple image.
Non-Patent Document 2 does not sufficiently classify background noise that is not a character string and characters.

  The present invention has been made in consideration of the above-described circumstances, and is an image processing apparatus and image that can accurately determine a color image or a grayscale image in which characters or images other than characters are mixed into a plurality of classes. An object of the present invention is to provide a processing method, a program for executing the functions of the image processing apparatus, and a computer-readable recording medium on which the program is recorded.

In order to solve the above-described problem, the invention according to claim 1 is characterized in that in the image processing apparatus that discriminates image information by a discriminator that has learned based on learning data, a feature amount extraction unit that extracts a feature amount from the image information. A feature calculation unit that calculates a feature that is a combination of the feature amounts extracted by the feature amount extraction unit, a feature amount calculated by the feature calculation unit, and a feature amount extracted by the feature amount extraction unit Learning means for learning the discriminator, collating means for applying the teacher data to the discriminator learned by the learning means to collate the discrimination result with an ideal discrimination result given from the outside, and collation in the collating means And an optimization unit that changes a combination method of the feature amounts in the feature calculation unit based on the result.
According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the optimization unit uses a method of combining feature amounts in the feature calculation unit by cross-validation analysis based on a collation result in the collation unit. It is characterized by changing optimally.

  According to a third aspect of the present invention, in an image processing apparatus that discriminates image information by a discriminator that has been learned based on learning data, a feature amount extraction unit that extracts a feature amount from image information, and the feature amount extraction unit extracts the feature amount. Obtaining a projection axis from the feature quantities of the plurality of images obtained, converting the feature quantity into a kernel feature quantity by the projection axis, and classifying the image information based on the kernel feature transformed by the feature transformation means. And classifying means for learning a discriminator for discriminating an image for each category classified by the classification means.

According to a fourth aspect of the present invention, in the image processing apparatus according to the first, second, or third aspect, the feature amount is acquired from a row candidate in the image.
According to a fifth aspect of the present invention, in the image processing apparatus according to the fourth aspect, the row candidates are obtained by integrating pixels that are similar in color to each other as a connected component and integrating a circumscribed rectangle of the connected component. And
According to a sixth aspect of the present invention, in the image processing apparatus according to the fifth aspect, the feature relating to the connected component is the feature amount.
According to a seventh aspect of the present invention, in the image processing apparatus according to the fourth or sixth aspect, the acquired feature amount moment is used as the feature amount.
According to an eighth aspect of the present invention, in the image processing apparatus according to the fifth aspect, when the row candidate has a small degree of color difference from other connected components in the vicinity of the connected component, both are the same. It is characterized by integrating as if it belonged to a line candidate.
According to a ninth aspect of the present invention, in the image processing apparatus according to the eighth aspect, when the other connected components in the vicinity are integrated, the row direction is assumed to be vertical or horizontal and exists in the row direction. Only the connected component is regarded as an object of integration.
According to a tenth aspect of the present invention, in the image processing apparatus according to the eighth or ninth aspect, the color difference is calculated using an average color of pixels constituting a connected component.

According to an eleventh aspect of the present invention, in the image processing apparatus according to the fourth aspect, the row candidates extract continuous pixels having close lightness as connected components.
According to a twelfth aspect of the present invention, in the image processing apparatus according to the eleventh aspect, when the row candidate has a small difference in lightness between other connected components in the vicinity of the connected component, both of them are the same row candidate. It is characterized by integrating as if belonging to.
According to a thirteenth aspect of the present invention, in the image processing apparatus according to the twelfth aspect, when the other connected components in the vicinity are integrated, the row direction is assumed to be vertical or horizontal, and exists in the row direction. Only the connected component is regarded as an object of integration.
According to a fourteenth aspect of the present invention, in the image processing apparatus according to the twelfth or thirteenth aspect, the lightness difference is calculated using an average lightness of pixels constituting a connected component.

  According to a fifteenth aspect of the present invention, in the image processing device according to any one of the first to fourteenth aspects, the feature amount extraction unit generates an image having a low resolution, and then generates a feature amount from the image having the low resolution. Is extracted.

  The invention according to claim 16 is an image processing method for discriminating image information by a discriminator learned based on learning data. A feature amount extraction step for extracting feature amounts from image information, and an extraction by the feature amount extraction step. A feature calculation step for calculating a feature that is a combination of the feature amounts, a learning step for learning a discriminator based on the feature amount calculated in the feature calculation step and the feature amount extracted in the feature amount extraction step; A matching step of applying the teacher data to the discriminator learned in the learning step to collate the discrimination result with an ideal discrimination result given from the outside, and based on the collation result in the collation step, in the feature calculation step And an optimization step of changing a combination method of feature amounts.

  The invention according to claim 17 is an image processing method for discriminating image information by a discriminator learned based on learning data. A feature amount extraction step for extracting feature amounts from image information, and an extraction by the feature amount extraction step. Obtaining a projection axis from the feature quantities of the plurality of images obtained, converting the feature quantity into a kernel feature quantity using the projection axis, and image information based on the kernel feature transformed by the feature transformation process. The method includes a classification step of classifying into categories, and a category-specific learning step of learning a discriminator that discriminates an image for each category classified in the classification step.

The invention according to claim 18 is a program for causing a computer to execute the function of the image processing apparatus according to any one of claims 1 to 15.
A nineteenth aspect of the present invention is a computer-readable recording medium on which the program according to the eighteenth aspect is recorded.

  According to the present invention, a color image or a grayscale image in which characters and images other than characters are mixed can be discriminated into a plurality of classes with high accuracy.

Hereinafter, preferred embodiments of an image processing apparatus according to the present invention will be described with reference to the drawings.
The image processing apparatus according to the present invention includes a learning portion that increases the discrimination accuracy of the discriminator using learning data, and a discrimination portion that discriminates an input image using the discriminator. Hereinafter, the learning part and the discrimination part will be described in detail in order.

<Embodiment 1>
In the first embodiment, learning data and teacher data are prepared by the user for learning of the discriminator.

(A) Learning Part FIG. 1 is a block diagram showing a functional configuration of a learning part in the first embodiment. In the figure, the learning part includes feature quantity extraction means 10, feature calculation means 20, learning means 30, collation means 40, optimization means 50 and learning data storage means 11, teacher data storage means 12, feature quantity storage means 13, and discrimination. Unit data storage means 14.

The learning data storage means 11 and the teacher data storage means 12 store a plurality of sets of image data used for learning of the discriminator and the correct answer of the discrimination result.
The feature amount extraction unit 10 extracts image data one by one from the learning data storage unit 11, extracts the feature amount from the image, and stores it in the feature amount storage unit 13 in association with the image data.
The feature amount extracted here may be a feature amount used in general image processing such as the color, size, and number of colors of an image. It is desirable to extract a plurality of feature quantities to be extracted.
Further, the feature quantity extraction unit 10 similarly extracts image data one by one from the teacher data storage unit 12, extracts the feature quantity from the image, and stores it in the feature quantity storage unit 13 in correspondence with the image data. . Here, the feature quantity for each image stored in the feature quantity storage unit 13 also distinguishes between learning data and teacher data.

  Furthermore, if feature values are extracted after converting the image data into low-resolution image data, the processing time required for feature value extraction can be reduced, and a certain color is a set of fine points (each color component dot). It is also possible to reduce the adverse effects of halftone dots that are expressed and tend to be noise during feature quantity extraction.

The feature calculation means 20 takes out the feature quantities extracted from the learning data stored in the feature quantity storage means 13 for each image and combines them by, for example, weighting and summing each feature quantity to calculate a new feature quantity. Then, it is stored in the feature amount storage means 13 in association with the image data.
Furthermore, the feature calculation means 20 similarly extracts the feature quantities extracted from the teacher data stored in the feature quantity storage means 13 for each image, and combines them by weighting and summing the feature quantities, for example. The feature amount is calculated and stored in the feature amount storage unit 13 in association with the image data.

The learning unit 30 learns the discriminator by extracting, for each image, the feature amount created in combination with the feature amount extracted from the learning data stored in the feature amount storage unit 13. For this discriminator, for example, a multilayer neural network or a support vector machine is used.
When learning for one image is completed, various parameters of the discriminator are temporarily stored and used as parameters of the discriminator to be learned by the next learning data.

  The matching unit 40 extracts, for each image, the feature quantity extracted from the teacher data stored in the feature quantity storage unit 13 and the new feature quantity created in combination, and uses the discriminator learned by the learning unit 30. Apply. The correct answer rate is obtained by counting whether the discrimination result matches the given correct answer, and the combination method, various parameters of the discriminator, and the correct answer rate are associated and temporarily stored.

The optimization unit 50 appropriately changes the combination method of the feature amounts by the feature calculation unit 20 and selects the best matching result. For example, when using the weighted sum for the combination method, the weight value is reset at random, the sum based on the weight value is obtained from the feature amount extracted from the learning data storage means 11, the discriminator is re-learned, and the teacher The data storage means 12 is applied to a discriminator to obtain a correct answer rate.
The optimizing means 50 repeats this operation a predetermined number of times, adopts a combination method (for example, the weight value is the final weight) and the discriminator at that time, which give the highest accuracy rate, and sends the discriminator data storage means 14 to the discriminator data storage means 14. The combination method and various parameters of the discriminator are stored.

Next, the processing flow of the learning part in the first embodiment will be described with reference to the flowchart of FIG.
Image data is extracted from the learning data storage unit 11 one by one, feature amounts are extracted from the images, stored in the feature amount storage unit 13 in correspondence with the image data, and similarly one from the teacher data storage unit 12. The image data is taken out one by one, the feature quantity is extracted from the image, and stored in the feature quantity storage means 13 in correspondence with the image data (step S1). The feature amount extracted here may be a feature amount used in general image processing such as the color, size, and number of colors of an image.

  The feature quantity extracted from the learning data and the teacher data stored in the feature quantity storage unit 13 is extracted for each image, for example, combined by weighting and summing each feature quantity, and calculating a new feature quantity. The data is stored in the feature amount storage means 13 in association with the data (step S2).

  The feature quantity created in combination with the feature quantity extracted from the learning data stored in the feature quantity storage means 13 is extracted for each image to learn the discriminator (step S3). For this discriminator, for example, a multilayer neural network or a support vector machine is used.

  The feature quantity created in combination with the feature quantity extracted from the teacher data stored in the feature quantity storage means 13 is extracted for each image, applied to the learned discriminator, and the discrimination result matches the correct answer. After applying the discriminator to all teacher data, the correct answer rate is obtained, and the combination method, various parameters of the discriminator and the correct answer rate are associated and temporarily stored (step S4).

  If learning of the determinator has not been performed a predetermined number of times (NO in step S5), the combination method of feature quantities is changed (step S6), and the process returns to step S2 in order to learn the determinator again by this combination method. Here, when the weighted sum is used in the combination method, the weight value is reset at random.

  On the other hand, if the learning of the determiner has been performed a predetermined number of times (YES in step S5), the best matching result is selected, and the combination method and the determiner at this time are stored in the determiner data storage unit 14. (Step S7), and the process ends. Here, when the weighted sum is used for the combination method, the weight giving the highest accuracy rate and various parameters of the discriminator are stored.

(B) Discrimination Part FIG. 3 is a block diagram showing the functional configuration of the discrimination part in the first embodiment. In the figure, the discrimination part is composed of a feature amount extraction means 10, a feature calculation means 20, a discrimination means 60, and a discriminator data storage means 14. The same functions as those in FIG. 1 are denoted by the same reference numerals and description thereof is omitted.

  The feature amount extraction unit 10 extracts a feature amount from the input image data. As the feature amount extracted here, the feature amount used in general image processing such as the color and size of the image and the number of colors may be used as described above. Further, when the feature quantity is extracted after the discriminator is learned, the feature quantity is extracted after converting the image data into the low resolution image data.

The feature calculation unit 20 calculates a new feature amount by combining the feature amounts by the combination method stored in the discriminator data storage unit 14. If the combination method is the weighted sum of feature amounts, the weight stored in the discriminator data storage unit 14 is extracted to calculate the sum of feature amounts.
The discriminating unit 60 applies the feature amount and the feature amount obtained by combining the feature amounts to the discriminator stored in the discriminator data storage unit 14 and outputs a discrimination result.

  By configuring the first embodiment as described above, it is possible to improve the discrimination accuracy by creating a new feature by combining feature amounts and updating the combination method optimally by supervised learning.

<Embodiment 2>
In the second embodiment, the above-described first embodiment is performed by cross-validation. A set of image data is divided into learning data and teacher data, and the learning method is changed by changing the dividing method. Repeat to get the best discriminator.

(A) Learning Part FIG. 4 is a block diagram showing a functional configuration of the learning part in the second embodiment. In the figure, the learning part is composed of a feature quantity extraction means 10, a feature calculation means 20, a learning means 30, a collation means 40, an optimization means 50, a division means 70, an image data storage means 15, and a division table 16. . The same functions as those in FIG. 1 are denoted by the same reference numerals and description thereof is omitted.

  The image data storage means 15 stores a plurality of image data as learning data and teacher data, and is generated by combining data items (file ID, file name, image data, correct value, feature value, feature value). New feature amount) (see FIG. 5).

  The dividing unit 70 divides the image data stored in the image data storage unit 15 into a predetermined number (for example, 3 groups) of groups so that the number of image data included in each group is substantially the same. 161, and a combination is generated so that these groups become learning data and teacher data, and stored in the learning data set table 162.

  For example, if the number of groups is 3, group assignment to learning data is (Group 1), (Group 2), (Group 3), (Group 1 and Group 2), (Group 1 and Group 3), Six combinations of (Group 2 and Group 3) are conceivable.

The group table 161 is a table that stores, for each group, a list of group IDs and file IDs of image data belonging to the group (see FIG. 6).
The learning data set table 162 is a table that stores combinations and learning results when each group is divided into learning data and teacher data (see FIG. 7).

The learning data set table 162 includes a combination ID, a list of group IDs belonging to the learning data, a list of group IDs belonging to the teacher data, and a combination method of feature amounts, various parameters of the discriminator, and teacher data that are learning results. It consists of the number of correct answers and the number of teacher data.
The group file correspondence table 161 and the learning data set table 162 are temporarily recorded as a division table 16 in a storage device such as a memory or a hard disk.

  The feature amount extraction unit 10 extracts feature amounts for all image data stored in the image data storage unit 15 and updates the image data storage unit 15 in correspondence with the image data. As the feature amount, for example, the color and size of an image used in general image processing, the number of colors, and the like are extracted as in the first embodiment. Alternatively, the feature amount may be extracted after the image data is converted into low-resolution image data.

The optimization means 50 searches the learning data stored in the learning data set table 162 for an unprocessed set, and for the set (hereinafter referred to as set A), a combination method of feature quantities, the number of correct answers, and a teacher Initialize the number of data.
For example, when the feature value combination method uses the weighted sum of feature values described in the first embodiment, an initial value of the weight value is set.

  The feature calculation unit 20 combines the feature amounts of all the image data stored in the image data storage unit 15 by the combination method specified in the set A, and stores the image data as new feature amounts corresponding to the image data. The means 15 is updated.

The learning means 30 refers to the learning data set table 162 to extract the group ID of learning data belonging to the set A, and refers to the group table 161 to identify image data belonging to this group ID (hereinafter referred to as image data B). Called).
Next, the discriminator is learned using the feature quantity corresponding to the image data B and the new feature quantity with reference to the image data storage means 15. For this discriminator, for example, a multilayer neural network or a support vector machine is used.
When learning is completed for one image data B, various parameters of the learned discriminator are stored in the learning data set table 162 in association with the set A.
The classifier is further trained using all other learning data belonging to the set A.

The collation means 40 refers to the learning data set table 162, extracts the group ID of the teacher data belonging to the set A, and refers to the group table 161 to identify the image data belonging to this group ID (hereinafter referred to as image data C Called).
Next, the feature quantity corresponding to the image data C and the new feature quantity are applied to the discriminator corresponding to the set A with reference to the image data storage means 15. If the determination result matches the correct answer, the number of correct answers is counted and the number of teacher data is counted up by one.

When these operations are completed, the optimization unit 50 is activated again to check whether all combinations of learning data registered in the learning data set table 162 have been processed.
When an unprocessed combination is found, the optimization unit 50 regards the found combination as the above-described set A, changes the combination method of the feature values appropriately, and determines the classifier, the number of correct answers, and the number of teacher data. Initialization is performed to newly learn the classifier. For example, when the weighted sum is used for the combination method, the combination method is changed by resetting the weight value at random.

  Further, when all the combinations of learning data are processed, the optimization means 50 obtains the correct answer rate = (number of correct answers / number of teacher data) for all registered in the learning data set table 162. The combination method that gives the highest accuracy rate and the discriminator at that time are adopted, and the adopted combination method and various parameters of the discriminator are stored in the discriminator data storage means 14.

Next, the processing flow of the learning part in the second embodiment will be described with reference to the flowchart of FIG.
First, the image data stored in the image data storage means 15 is divided into a predetermined number of groups and stored in the group table 161 (see FIG. 6) so that the number of image data included in each group is substantially the same. Then, a combination is generated so that these groups become learning data and teacher data, and stored in the learning data set table 162 (see FIG. 7) (step S10).

  A feature amount (for example, image color, size, number of colors, etc.) is extracted from all the image data stored in the image data storage unit 15, and the image data storage unit 15 is updated corresponding to the image data.

  The learning data stored in the learning data set table 162 is searched for an unprocessed set (hereinafter referred to as set A) (step S12). If there is an unprocessed set A (NO in step S13), For the set, the combination method of feature values, the number of correct answers and the number of teacher data are initialized (step S14).

  The feature amounts of all the image data stored in the image data storage unit 15 are combined by the combination method specified in the set A, and the image data storage unit 15 is updated corresponding to the image data as a new feature amount ( Step S15).

  Referring to the learning data set table 162 and the group table 161, the discriminator is learned using the feature amount corresponding to the learning data belonging to the set A and the new feature amount, and various parameters of the learned discriminator are associated with the set A. And stored in the learning data set table 162 (step S16).

  With reference to the learning data set table 162 and the group table 161, the feature quantity corresponding to the teacher data belonging to the set A and the new feature quantity are applied to the discriminator corresponding to the set A, and the discrimination result matches the correct answer. When doing so, the number of correct answers is counted and the number of teacher data is counted up by one (step S17).

  When these operations are completed, it is confirmed whether all combinations of learning data registered in the learning data set table 162 have been processed. If an unprocessed combination is found (NO in step S13), the found combination is described above. The combination method of the feature values is appropriately changed, the classifier, the number of correct answers, and the number of teacher data are initialized (step S14), the classifier is newly learned, and the correct answer is determined by the teacher data. The calculation of the number of times (steps S15 to S17) is repeated.

  On the other hand, when all the combinations of learning data are processed (YES in step S13), the correct answer rate = (number of correct answers / number of teacher data) is obtained for all registered in the learning data set table 162. The combination method with the highest accuracy rate and the discriminator at that time are adopted, the adopted combination method and various parameters of the discriminator are stored in the discriminator data storage means 14 (step S18), and the learning part Terminate the process.

(B) Discrimination part The discrimination part in the present embodiment is configured in the same manner as in the first embodiment, and thus the description thereof is omitted.

  By configuring the second embodiment as described above, it is possible to improve the discrimination accuracy by creating a new feature by combining feature amounts and optimally updating the combination method by supervised learning. At this time, learning data and teacher data can be selected in an optimal combination by cross-validation, and learning data without bias can be used, so that the discrimination accuracy can be further improved.

<Embodiment 3>
Next, in the third embodiment, it is considered that the first and second embodiments are used to accurately determine whether a character line candidate extracted from image data is really a character line.
For this purpose, learning data for learning of the discriminator is created by one of the following two methods.

(1) First method Image data constituting a character line and other image data are prepared. It is assumed that these image data are distinguished from each other as to whether the image data is a character line or not.
In the first embodiment, the user provides these image data separately as learning data and teacher data, and in the second embodiment, these image data may be provided as they are.
Further, a character line candidate extraction process (described later) is performed from the given image data, and a feature amount is extracted from the extracted character line candidate image and applied to each embodiment.

(2) Second Method When there are a plurality of character line candidates in one piece of image data, whether or not the user is a character line with respect to the character line candidate area extracted by performing the character line candidate extraction process. This instruction and the image data of the character line candidate area are recorded as a pair.
By applying this operation to a plurality of pieces of image data, learning data of the discriminator is created, and this learning data is divided into learning data and teacher data by a user or a cross-validation technique and applied to each embodiment.

  When using a discriminator, a character line candidate is extracted from the input image data, and a new feature quantity obtained by combining the feature quantity and the feature quantity is applied to the discriminator. To determine.

Next, the character line candidate extraction process will be described.
A well-known technique (for example, Unexamined-Japanese-Patent No. 2003-208568) can be applied to extraction of a character line candidate.
For example, if the colors of adjacent pixels in the horizontal direction are close to each other, run them by combining them as processing units, compare the colors of those that touch the vertical direction for these runs, and connect if the colors are close Integration as a component, and a circumscribed rectangle of the connected component of the integrated character is generated. In this way, a group of pixels that are character candidates can be extracted as one circumscribed rectangle. Here, various methods can be adopted as a method for determining that the colors are close. For example, the sum of squares of differences between color components (RGB, etc.) of pixel values is calculated, and this is calculated between pixels. Considering the degree of color difference, it is determined that this value is close to a case where this value is smaller than a predetermined value based on an experimental value or the like.

Next, adjacent circumscribed rectangles are integrated by determining the color similarity of adjacent circumscribed rectangles and determining the distance between the rectangles. The integrated circumscribed rectangle obtained by repeating this determination is extracted as a character line candidate.
Here, in the determination of the color similarity, when the average value of the pixels included in the circumscribed rectangle or the difference between the representative colors is smaller than a predetermined value, it is determined that they are similar. Thereby, it is possible to improve the integration accuracy of the character line candidates while suppressing the influence of the color unevenness of the pixels constituting the character line.

Further, in the above-described circumscribed rectangle integration process, a restriction may be provided depending on the direction of the character line (whether it is a vertically written sentence or a horizontally written sentence). For example, as preprocessing, the entire image data is subjected to the circumscribed rectangle integration processing as described above to generate a character line candidate group, and the majority of the character line candidates is determined in which direction the decision is made. The bounding rectangles are integrated only in the direction of the specified character line.
As a result, the line direction can be limited to vertical or horizontal by pre-processing, so that it is possible to improve the discrimination accuracy between the character line and the other.

  Further, when the circumscribed rectangles are integrated, the rectangles may be integrated according to the similarity based on the lightness in the rectangles. In this case, it is possible to effectively extract character line candidates even for a grayscale image.

  A circumscribed rectangle may be used as the feature quantity. In this case, the width or height of the circumscribed rectangle may be directly used as a feature amount, or a moment calculated from the feature amount may be used as a new feature amount. Alternatively, a feature amount such as the width or height of each connected component existing in the row may be obtained, and a moment may be calculated from the feature amounts of all the connected components in the row to obtain a new feature amount.

  In general, the n-th moment M (n) around the average μ can be calculated by the following equation 1.

  Here, x is the feature quantity, n is the moment order, E () is a symbol representing the average, and μ is the average value of the feature quantity x.

<Embodiment 4>
In the fourth embodiment, learning data is clustered in advance, and an optimal discriminator is obtained by learning for each cluster.

(A) Learning Part FIG. 9 is a block diagram showing a functional configuration of the learning part in the fourth embodiment. In the figure, the learning part is composed of a feature amount extraction means 10, a conversion axis derivation means 80, a feature conversion means 90, a classification means 100, a cluster-by-cluster learning means 110, an image data storage means 15, and a cluster-by-cluster discriminator data storage means 18. Composed. The same functions as those in FIGS. 1 and 4 are denoted by the same reference numerals and description thereof is omitted.

  The image data storage unit 15 stores a plurality of pieces of image data serving as learning data, and includes data items (file ID, file name, image data, correct value, feature value, converted feature value, class) (FIG. 10). reference).

The feature amount extraction unit 10 extracts feature amounts for all image data stored in the image data storage unit 15 and updates the image data storage unit 15 in correspondence with the image data. As this feature amount, for example, the color and size of an image used in general image processing, the number of colors, and the like are extracted in the same manner as in the above embodiments. Further, by extracting the feature amount after converting the image data into low-resolution image data, noise due to processing time and halftone dots can be reduced.
By using the feature amount extracted from the character line candidates as described in the third embodiment as the feature amount, it is expected that the accuracy of discrimination between the character line and other characters is further improved.

  The conversion axis deriving unit 80 performs kernel principal component analysis on the feature amount for all images stored in the image data storage unit 15 to obtain a projection axis. This kernel principal component analysis is a method that effectively calculates the principal component axis of a high-dimensional feature space that has been nonlinearly mapped. For example, Bernhard Scholkopf, Alexander Smola, Klaus-Robert Muller, “Nonlinear Component Analysis as a Kernel Eigenvalue Problem ”, Neural Computation, 10, pp.1299-1319, 1998). Kernels used for the kernel principal component analysis include a polynomial kernel (Polynomial Kernel), a Gaussian kernel (Gaussian Kernel), and a sigmoid kernel (Sigmoid Kernel).

This kernel principal component analysis has the following merits.
・ Visualization of data in the kernel feature space can be visualized.
-Expressive power varies depending on differences in kernel functions (Polynomial Kernel, Gaussian Kernel, Sigmoid Kernel, etc.) and parameter values (Polynomial Kernel degree, Gaussian Kernel distribution, etc.).
In particular, kernel principal component analysis using a degree 1 Polynomial Kernel is equivalent to linear principal component analysis of the input space.

  The feature conversion unit 90 applies the projection axis to the feature amounts of all the images stored in the image data storage unit 15 to project them to the kernel feature amount, and stores the image data corresponding to the image data. The means 15 is updated.

  The classification unit 100 clusters the kernel feature values of all the images stored in the image data storage unit 15 and updates the image data storage unit 15 so that the cluster corresponds to the image data. As this clustering method, a known k-means method or k-nearest neighbor method is used.

The cluster-by-cluster learning unit 110 takes out the image data stored in the image data storage unit 15 for each cluster, learns the discriminator for each cluster using the feature amount of the extracted image, and stores the discriminator data by cluster. The data is stored in the means 18. As the discriminator, a multilayer neural network or a support vector machine is effective.
The cluster-specific discriminator data storage means 18 also stores the projection axis derived by the conversion axis deriving means 80.

Next, the processing flow of the learning part in the fourth embodiment will be described with reference to the flowchart of FIG.
Feature amounts are extracted from all the image data stored in the image data storage unit 15, and the image data storage unit 15 is updated in correspondence with the image data (step S20).
A kernel principal component analysis is performed on the feature amount for all images stored in the image data storage unit 15 to obtain a projection axis (step S21).
The projection axis is applied to the feature quantities of all the images stored in the image data storage means 15 to project to the kernel feature quantities, and the image data storage means 15 is updated corresponding to the image data ( Step S22).

The kernel feature values of all the images stored in the image data storage unit 15 are clustered by using a known k-means method or k-nearest neighbor method, and the image data storage unit 15 is made to correspond to the image data. Is updated (step S23).
The image data stored in the image data storage unit 15 is extracted for each cluster, and a classifier composed of a multilayer neural network, a support vector machine, or the like is learned for each cluster and stored in the cluster-specific classifier data storage unit 18. At the same time, the derived projection axis is stored (step S24).

(B) Discrimination Part FIG. 12 is a block diagram showing the functional configuration of the discrimination part in the fourth embodiment. In the figure, the discriminating part is composed of a feature quantity extracting means 10, a feature converting means 90, a classifying means 100, a cluster discriminating means 120 and a cluster discriminator data storage means 18. About the same function as FIG. 9, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The feature amount extraction unit 10 extracts a feature amount from the input image data. As the feature amount extracted here, the feature amount used in general image processing such as the color and size of the image and the number of colors may be used as described above. Further, when the feature quantity is extracted after the discriminator is learned, the feature quantity is extracted after converting the image data into the low resolution image data.

The feature conversion unit 90 projects the feature amount using the projection axis stored in the cluster-specific discriminator data storage unit 18 and calculates a cluster feature amount.
The classification unit 100 clusters kernel feature amounts using, for example, a known k-means method or k-nearest neighbor method. For example, when the k-means method is used, the distance from the cluster center obtained in the learning part is calculated and classified into the nearest cluster.
The cluster-specific discriminating unit 120 refers to the cluster-specific discriminator data storage unit 18, applies the feature amount of the image to the discriminator corresponding to the cluster classified by the classification unit 100, and outputs a discrimination result.

Next, the flow of processing of the determination part in the fourth embodiment will be described with reference to the flowchart of FIG.
A feature amount (for example, the color and size of the image, the number of colors, etc.) is extracted from the input image data (step S30).
The feature quantity is projected by the projection axis stored in the cluster discriminator data storage means 18 to calculate the cluster feature quantity (step S31).
Using the k-means method or k-nearest neighbor method known in the art, the kernel feature is classified into the nearest cluster (step S32).
With reference to the cluster discriminator data storage means 18, the feature quantity of this image is applied to the discriminator corresponding to the cluster classified by the classification means 100, and the discrimination result is output (step S33).

  With the above configuration, it is possible to improve the discrimination accuracy by performing clustering on the feature quantity mapped to the nonlinear space and generating a discriminator for each cluster.

<Embodiment 5>
Next, in the fifth embodiment, as described in the third embodiment, it is considered that the fourth embodiment is used to accurately determine whether or not the character line candidate extracted from the image data is really a character line. .

(1) First method Image data constituting a character line and other image data are prepared. With respect to these image data, learning image data is created by distinguishing whether the image data is a character line or not.

(2) Second Method When there are a plurality of character line candidates in one piece of image data, is the user a character line for the character line candidate image extracted by performing the character line candidate extraction process? Whether or not, and this instruction and the image data of the character line candidate are recorded as a pair. By applying this operation to a plurality of pieces of image data, image data for learning of the discriminator is created.

  In the fourth embodiment, the character line candidate extraction process is performed from the image data created by the method (1) or (2), and the feature amount is extracted from the extracted image area of the character line candidate. Have the vessel learn.

<Embodiment 6>
Furthermore, the present invention is not limited only to the above-described embodiments. Each function constituting the image processing apparatus according to the above-described embodiment is programmed and written in advance on a recording medium such as a CD-ROM, and this CD-ROM is mounted with a medium driving device such as a CD-ROM drive. The object of the present invention can be achieved by mounting these programs on a computer and storing these programs in a memory or storage device of a computer and executing them.

  As a recording medium, a semiconductor medium (for example, ROM, nonvolatile memory card, etc.), an optical medium (for example, DVD, MO, MD, CD-R, etc.), a magnetic medium (for example, magnetic tape, flexible disk, etc.) Either may be sufficient.

  In addition, the case where an operating system, an application program, or the like performs part or all of the actual processing based on the instruction of the loaded program and the functions of the above-described embodiments are realized by the processing is also included.

In addition, when the above-mentioned program is stored in a storage device such as a magnetic disk of a server computer and downloaded from a user computer connected via a network and distributed, or distributed and distributed from a server computer, The storage device of this server computer is also included in the recording medium of the present invention.
In this way, by programming the function of the present invention, recording it on a recording medium and distributing it, it is possible to improve cost, portability and versatility.

  This embodiment is an experiment for recognizing a character line from a color document image in which characters and photographs are mixed, using the above-described embodiment.

(1) Feature extraction First, character line candidates are exhaustively extracted from a color document image. Character line candidates can be obtained by grouping connected components of the same color in the horizontal direction and the vertical direction one after another based on the similarity and position of the color components. Here, a character line candidate that is truly a character line is called Positive, and the others are called Negative. The character line candidates were previously determined by human eyes and labeled.

In addition, the following feature amounts are assigned to the character line candidates.
・ Special amount related to contrast.
-Features related to sparseness in the circumscribed rectangle of the connected component.
・ Features calculated based on the number of circumscribed rectangles.

(2) Feature space First, principal component analysis was performed to visualize the distribution of high-dimensional feature values in the feature space. After taking 2000 samples from Positive and Negative respectively and normalizing them, the principal component analysis was tried by two methods.

・ Calculate principal components from only Positive data, and project both Positive and Negative data based on the result.
・ A method to calculate principal components from only negative data and project both positive and negative data based on the result.

When the incremental change of the cumulative contribution rate by each method is plotted, the result shown in FIG. 14 is obtained.
As a result, it was found that the positive data is up to the fourth principal component and can explain the original high-dimensional feature space by about 80%. On the other hand, the negative data must be calculated up to the sixth principal component in order to explain about 80% of the original high-dimensional feature space. This means that the distribution of the positive data in the feature space can be explained with a relatively small principal component, and the variation in the multidimensional direction is relatively small.

  Then, when the negative data was projected onto the first principal component and the second principal component of the positive data, the separability between the positive cluster and the negative cluster could be confirmed (see FIG. 15). At this time, the cumulative contribution rate is 71.88%.

  Next, a scatter diagram when the positive data is projected onto the first principal component and the fourth principal component of the negative data (see FIG. 16), and a scatter when the positive data is projected onto the first principal component and the sixth principal component. The figure (refer FIG. 17) was cut out. As a result, we have confirmed that Negative has several clusters.

  This cluster can be roughly divided into three. From the factor loading of the original features, it was found that these clusters correspond to the shape of the character line candidates (that is, cluster 1: landscape, cluster 2: portrait, cluster 3: square).

  In the high-dimensional feature space, Positive exists in a relatively low-dimensional manner without scattering in multi-dimensional directions, Negative has a relatively large variation, and has a cluster structure based on the shape of character line candidates. I was able to confirm that.

(3) Learning Next, evaluation was performed using several supervised learning algorithms depending on the separability of Positive and Negative clusters after principal component analysis.
208 samples of color document images were prepared for learning, and 19588 entries for Positive and 72027 entries for Negative were obtained. In addition, 41 color document images were prepared for teachers, and 2730 entries for positive and 35495 entries for negative were obtained (see Table 1).

  Also, before learning, high-dimensional feature values are feature values with different ranges, so in order to accurately capture the feature distance, each feature value was normalized so that the average of the feature values was 0 and the variance was 1. .

(3-1) Multi-level perceptron (MLP)
First, learning was performed with a three-layer perceptron having one intermediate layer in addition to the input layer and the output layer. By the 10 fold cross validation method for the learning data set 30000 (Positive: 15000, Negative: 15000), the optimal number of hidden nodes, which is a parameter of the 3-layer perceptron, is 80.

After learning with the learning data set and evaluating with the teacher data set, if the output threshold is set to 0.5, Positive can be recognized at 90.77%, and for all character line candidates (Positive + Negative) Was recognized at 95.60%.
Further, it was found that if the positive recognition accuracy is increased by drawing ROC (receiver operator characteristics) curves by changing the threshold value, the negative recognition accuracy is lowered (see FIG. 18). If the MLP output threshold is set to 0.1 in order to make the positive recognition accuracy 95%, the negative recognition accuracy is reduced to 87%.

(3-2) Support Vector Machine (SVM)
Next, we learned with a support vector machine and evaluated it. It should be noted that a support vector classifier (νSVC) in which the parameter value range is determined by 0 <ν <1 is used instead of the C parameter for the support vector machine. The kernel function used was Polynomial Kernel shown in (Equation 2) below.

The parameter d of Polynomial Kerne1 and the parameter ν of νSVC are determined by 10 fold cross validation as in the case of the multilayer perceptron, and d = 8 and ν = 0.12 are used.
As a result, it was possible to recognize 91.81% of data Positive that is a truly character line, and 95.49% of all character line candidates (Positive + Negative).

(3-3) Mixture of Experts (MoE)
By cluster analysis, it was found that character line candidates can be divided into three: a horizontally long character line candidate, a vertically long character line candidate, and a square character line candidate having only one character.
As a result of error analysis of MLP and SVM, it was found that a horizontal character line candidate can be correctly recognized, but a square character line candidate is often erroneously recognized. This is because the prior probability that a square object is a character string is lower than a horizontally long object, and the current feature value tends to determine that a square object is not a character string. It is.

  Therefore, the MoE model, which is one of the divide-and-conquer algorithms that divide and capture the feature space, was adopted. MoE is composed of three nodes: Expert Network responsible for output for input x, Gating Network responsible for appropriate weighting for each Expert Network for input x, and Join Nodes that combine the outputs from each Expert Network. (FIG. 19).

All learning data sets are set as learning data sets for the square model when the height (H) and width (W) of the character line candidates are (1-H / W) ² <0.1, and H / W> When it is 1, it is set as a learning data set for a vertically long model, and otherwise it is set as a learning data set for a horizontally long model.

Furthermore, a new feature quantity “outline score” that can be assigned to each character line candidate shape (horizontal model, vertical model, square model) has been added. This “outline score” is the height h and width of the circumscribed rectangle, the height of the character line candidate is H, the width is W, the number of connected characters is a, the area of the circumscribed rectangle is A, and the variables v ₁ ,. , V ₆ is defined as follows,

v ₁ = h / H,
v ₂ = h / w,
v ₃ = w / h,
v ₄ = w / W,
v ₅ = a / A,
v ₆ = A / a

If the weighting parameters are w ₁ ,..., W ₆ , the weighted sum S = Σv _i w _{i is} applied to the circumscribed rectangle forming the character line candidate.
Therefore, it was made a special amount of character line candidates.

The weights w _i are randomly determined so that the recognition accuracy is highest when 4-fold cross validation is performed, and learning data sets assigned to each model (square, landscape, portrait) are learned by νSVC. At this time, the parameters (d and ν) of each model were determined by 10 fold cross validation.

  When recognizing, Gating Network classifies the image data of the character line candidates into one of the character line candidate shapes (horizontal, vertical, square), and finally outputs the output from Expert Network corresponding to this character line candidate shape. The output was changed. As a result, it was possible to recognize 91.6% of data Positive which is a truly character string, and 96.46% of all character line candidates (Positive + Negative).

  As described above, it is possible to recognize a character line region with high accuracy by using a supervised learning method such as SVM or MoE after image processing and feature extraction for a complex color document.

FIG. 3 is a block diagram illustrating a functional configuration of a learning part in the first embodiment. 3 is a flowchart showing a flow of processing of a learning part in the first embodiment. FIG. 3 is a block diagram illustrating a functional configuration of a determination part in the first embodiment. FIG. 10 is a block diagram illustrating a functional configuration of a learning part in the second embodiment. It is an example of the data structure of the image data storage means in Embodiment 2. It is an example of a data structure of the group table in Embodiment 2. It is an example of a data structure of the learning data set table in Embodiment 2. 10 is a flowchart showing a flow of processing of a learning part in the second embodiment. FIG. 10 is a block diagram illustrating a functional configuration of a learning part in the fourth embodiment. 10 is a data structure example of an image data storage unit in the fourth embodiment. 10 is a flowchart showing a flow of processing of a learning part in the fourth embodiment. FIG. 10 is a block diagram illustrating a functional configuration of a determination part according to a fourth embodiment. 10 is a flowchart showing a flow of processing of a determination part in the fourth embodiment. It is a graph which shows the incremental change of the cumulative contribution rate according to the method of two types of principal component analyses. It is a scatter diagram which shows the separability of Positive cluster and Negative cluster when negative data is projected on the 1st main component and 2nd main component of Positive data. It is a scatter diagram when Positive data is projected on the 1st principal ingredient and the 4th principal ingredient of Negative data. It is a scatter diagram when Positive data is projected on the 1st principal ingredient and the 6th principal ingredient. It is a graph which shows the relationship between a threshold value and recognition accuracy in MLP. It is a figure for demonstrating the structure of MoE.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Feature quantity extraction means, 20 ... Feature calculation means, 30 ... Learning means, 40 ... Collation means, 50 ... Optimization means, 60 ... Discrimination means, 70 ... Dividing means, 11 ... Learning data storage means, 12 ... Teacher data Storage means 13 ... Feature value storage means 14 ... Discriminator data storage means 15 ... Image data storage means 16 ... Division table 161 161 Group table 162 Learning data set table 80 Conversion axis derivation means 90 ... Feature conversion means, 100... Classification means, 110... Cluster learning means, 120.

Claims (19)

  In an image processing apparatus that discriminates image information by a discriminator that has learned based on learning data, a feature amount extraction unit that extracts a feature amount from image information and a feature that is a combination of the feature amount extracted by the feature amount extraction unit A feature calculation means for calculating the classifier, a learning means for learning the discriminator based on the feature quantity calculated by the feature calculation means and the feature quantity extracted by the feature quantity extraction means, and a discriminator learned by the learning means A collation unit that applies the teacher data to collate a discrimination result with an ideal discrimination result given from the outside, and an optimum that changes a combination method of feature amounts in the feature calculation unit based on the collation result in the collation unit And an image processing apparatus.   The image processing apparatus according to claim 1, wherein the optimization unit optimally changes a combination method of feature amounts in the feature calculation unit by cross-validation analysis based on a matching result in the matching unit. Image processing device.   In an image processing apparatus that discriminates image information by a discriminator that has learned based on learning data, a feature amount extracting unit that extracts a feature amount from image information, and a feature amount of a plurality of images extracted by the feature amount extracting unit Obtaining a projection axis, converting the feature quantity into a kernel feature quantity by the projection axis, classifying means for classifying image information into categories based on the kernel feature converted by the feature conversion means, and the classification An image processing apparatus comprising: a category-based learning unit that learns a discriminator that determines an image for each category classified by the unit.   The image processing apparatus according to claim 1, wherein the feature amount is acquired from a row candidate in the image.   The image processing apparatus according to claim 4, wherein the row candidates are obtained by integrating pixels having continuous similar colors as a connected component and integrating a circumscribed rectangle of the connected component.   6. The image processing apparatus according to claim 5, wherein a feature relating to the connected component is the feature amount.   7. The image processing apparatus according to claim 4, wherein a moment of the acquired feature quantity is used as the feature quantity.   6. The image processing apparatus according to claim 5, wherein when the row candidate has a small color difference from other connected components in the vicinity of the connected component, they are regarded as belonging to the same row candidate and integrated. An image processing apparatus.   9. The image processing apparatus according to claim 8, wherein when other connected components in the vicinity are integrated, the row direction is assumed to be vertical or horizontal, and only the connected components existing in the row direction are to be integrated. An image processing apparatus characterized by being regarded.   The image processing apparatus according to claim 8, wherein the color difference is calculated using an average color of pixels constituting a connected component.   The image processing apparatus according to claim 4, wherein the row candidate extracts continuous pixels having lightness as connected components.   12. The image processing device according to claim 11, wherein when the line candidate has a small difference in brightness of other connected components in the vicinity of the connected component, both are considered to belong to the same line candidate and integrated. A featured image processing apparatus.   The image processing apparatus according to claim 12, wherein when other connected components in the vicinity are integrated, the row direction is assumed to be vertical or horizontal, and only the connected components existing in the row direction are to be integrated. An image processing apparatus characterized by being regarded.   14. The image processing device according to claim 12, wherein the brightness difference is calculated using an average brightness of pixels constituting a connected component.   15. The image processing apparatus according to claim 1, wherein the feature amount extraction unit generates an image with a low resolution and then extracts a feature amount from the image with the low resolution. An image processing apparatus.   In an image processing method for discriminating image information by a discriminator that has been learned based on learning data, a feature amount extraction step that extracts a feature amount from image information and a feature that is a combination of the feature amount extracted by the feature amount extraction step A feature calculation step of calculating a classifier, a learning step of learning a discriminator based on the feature amount calculated in the feature calculation step and the feature amount extracted in the feature amount extraction step, and a discriminator learned in the learning step A collation process for applying a teacher data to collate a discrimination result with an ideal discrimination result given from the outside, and an optimal method for changing a combination method of feature quantities in the feature calculation step based on the collation result in the collation step And an image processing method.   In an image processing method for discriminating image information by a discriminator learned based on learning data, a feature amount extraction step for extracting feature amounts from image information, and feature amounts of a plurality of images extracted by the feature amount extraction step Obtaining a projection axis, converting the feature quantity into a kernel feature quantity by the projection axis, a classification process for classifying image information into categories based on the kernel feature converted by the feature conversion process, and An image processing method comprising: a learning step by category for learning a discriminator that discriminates an image for each category classified by the classification step.   A program for causing a computer to realize the functions of the image processing apparatus according to any one of claims 1 to 15.   The computer-readable recording medium which recorded the program of Claim 18.