JP6521440B2 - Neural network and computer program therefor - Google Patents

Description

  The present invention relates to machine learning, and more particularly to a neural network capable of performing inference with high accuracy even with a computer having a small computing resource.

  Machine learning is a learning method based on inductive reasoning that aims to acquire general rules or trends from a limited amount of cases. Mathematically, it is formulated as assuming that the case (training) data is sampled as a sample from the true data distribution, and estimating the data distribution of the extraction source globally. Data distribution can be regarded as a continuous function with a data space as a domain and a space expressing data likeness such as probability as a range. A neural network is one of the methods for approximating this continuous function from training data. Neural networks are widely used as multi-layered perceptrons (MLPs) or feed forward neural networks (FFNNs) in various identification tasks such as language models, speech recognition, and image recognition.

  Any continuous function or data distribution can be represented in a two-layer neural network using a logistic sigmoid function as an activation function. However, in networks with as few as two layers, only local distributions such as individual training data points are learned, and there is a tendency for overfitting to occur where global distribution can not be obtained. This situation is shown in FIG. FIG. 1 assumes that there are two types of data in a two-dimensional data space, and their training data (indicated by x and ○) are obtained. When identifying these two types of data as a task of a neural network, it is necessary to find a discrimination curve for identifying these two types of data in the original data space. However, if the number of parameters assumed is too large, the discrimination curve often overfits the training data. If the discrimination curve obtained by overfitting is, for example, the discrimination curve of FIG. 1, this discrimination curve 30 has very high discrimination performance for training data, sometimes giving complete discrimination results, but discrimination performance for data other than training data. On the contrary, the (generalization performance) deteriorates. On the other hand, the discrimination curve 32 obtained by proper training does not necessarily give complete discrimination results for all test data, but data with relatively high accuracy is obtained even when data other than training data is given. Can be identified.

  In order to suppress overfitting, a method called regularization (Non-Patent Document 1, Non-Patent Document 2) is widely used in machine learning in general. The regularization is a method of reducing the number of parameters affecting the approximation or reducing the value of the parameters when the discriminative model learns to approximate training data. This approach is based on the assumption that the original distribution from which training data is extracted has a smooth shape. That is, regularization is based on the assumption that the original distribution is a function composed of few and small valued parameters. Generally, it is used for the purpose of setting the values of many parameters to 0 or near 0 by adding L2 norm or L1 norm of parameters to the objective function to be minimized. That is, the generalization performance is improved by imposing the restriction that the number of parameters used for inference is substantially limited by regularization.

  Along with regularization, there is a concept of improving generalization performance from the viewpoint of calculation structure (function definition) of a model. One of the methods is a multilayered neural network called deep learning. A neural network consisting of many layers is said to have the property of capturing local data distribution in the layer close to the input, and capturing global distribution by combining local data distribution as it goes to the output layer. (Non-Patent Document 3). In some identification tasks such as image recognition, deep learning has achieved generalization performance that greatly surpasses conventional methods (Non-Patent Document 4).

  FIG. 2 shows a configuration example of a multilayered neural network (deep neural network: DNN) 50. Referring to FIG. 2, DNN 50 includes an input layer 52, an intermediate layer 54 composed of a plurality of hidden layers, and an output layer 56. In this example, the middle layer 54 includes five hidden layers. An error back propagation method is used for learning of DNN50. That is, training data is given to the input layer 52, and the predicted output for the training data is obtained from the output layer 56 through the intermediate layer 54. The error between the output and the correct data is reversely given from the output side, and the weight and bias of coupling between each node are updated so as to minimize a predetermined error function.

  Conventionally, logistic sigmoid functions, hyperbolic tangents, etc. have been widely used as activation functions for each layer. Recently, however, various functions (5 and 6) have been proposed for activation functions of each layer, assuming multi-layering, and it is shown that generalization performance is improved by these functions. .

Evgeniou, T., Pontil, M., & Poggio, T. (2000). Regularization networks and support vector machines. Advances in computational mathematics, 13 (1), 1-50. Girosi, F., Jones, M., & Poggio, T. (1995). Regularization theory and neural networks architectures. Neural computation, 7 (2), 219-269. Montufar, G., Pascanu, R., Cho, K. and Bengio, Y. (2014). On the Number of Linear Regions of Deep Neural Networks. Http://arxiv.org/abs/1402.1869 Schmidhuber, J. (2015). Deep Learning in Neural Networks: An Overview. Neural Networks, Vol 61, pp 85-117. (Http://arxiv.org/abs/1404.7828) Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep sparse rectifier networks. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR W & CP Volume. Vol. Goodfellow, I., Warde-farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013). Maxout Networks. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pages 1319-1327.

  As described above, the prior art shows improvement in generalization performance by multi-layering. However, there is a problem that the amount of calculation increases at the same time. Therefore, large-scale computing devices are still required to improve performance. In addition, although the computational efficiency of each layer is improved due to the progress of large-scale parallelization of computer hardware such as the spread of general-purpose GPU computers, in the multi-layer model, the calculation of the next layer is not completed until the calculation on the input side is completed. There is a restriction that it is possible. Therefore, there is a problem that the waiting time inevitably increases in proportion to the increase in the number of layers due to the multi-layering, and the calculation efficiency at the time of training as well as at the time of prediction decreases.

  Therefore, an object of the present invention is to provide a high generalization performance neural network that can be predicted in a short time.

  A neural network according to a first aspect of the present invention includes an input layer having a plurality of input nodes, and a non-linear function layer having an input connected to receive an output from the input layer. The non-linear function layer includes a plurality of hidden layers each capable of approximating any independent component, each connected to receive an output from a plurality of input nodes of the input layer. Each of the plurality of hidden layers has an input connected to each of the plurality of input nodes in a weighted manner, and is identical to a plurality of neural cell elements using a periodically differentiable function as an activation function. And an output aggregation element connected so as to receive the outputs of the plurality of neural cell elements in the hidden layer in a weighted manner and using a function that is differentiable as an activation function.

  Preferably, the periodic function is a cosine function or a sine function or a combination thereof, or a piecewise linear approximated cosine function or a sine function.

  More preferably, the neural network is trained using L2 regularization on the input side weight parameters of each of the plurality of hidden layers.

  The neural network may be trained using the quasi-Newton method.

  More preferably, the activation function of the output aggregation element may be an identity function, ie a function that outputs the same value as the given input.

  The neural network may further include an output layer connected to receive the outputs of the plurality of hidden layer output aggregation elements. The output layer includes a plurality of output neural cell elements each connected to receive the outputs of the plurality of hidden layer output aggregation elements in a weighted manner.

  The activation function of the output aggregation element may be a softmax function.

  A computer program according to a second aspect of the present invention causes a computer to function as any of the neural networks described above.

It is a figure which shows the relationship of the data distribution and identification function in two-dimensional space for demonstrating the problem of the overfitting in machine learning. It is a schematic diagram of the neural network for demonstrating the problem of the conventional deep neural network. FIG. 1 schematically shows a schematic configuration of a neural network according to an embodiment of the present invention. It is a graph which shows an example of the objective function when not adding a regular term to an error. It is a graph which shows an example of an objective function when a regular term is added to error. It is a block diagram of the speech recognition device using the acoustic model by the neural network concerning one embodiment of the present invention. In one embodiment of this invention, it is a graph which shows the improvement of the generalization performance in handwritten character recognition when regularization is carried out. In one embodiment of the present invention, it is a graph which shows generalization performance in handwriting character recognition when not regularizing. FIG. 1 is an external view of a computer system for realizing a character recognition device according to an embodiment of the present invention. It is a block diagram which shows the hardware constitutions of a computer shown in FIG.

  In the following description and the drawings, the same parts are given the same reference numerals. Therefore, detailed description about them will not be repeated. In the following description of the embodiment, the neural network is referred to as "NN".

[Basic way of thinking]
In machine learning, as a framework for obtaining a continuous function approximating training data, one embodiment of the present invention described below uses a network structure representing a frequency domain in discrete Fourier transform of the continuous function.

  FIG. 3 shows the structure of the NN used in the present embodiment, taking a function that approximates two label distributions as an example. Referring to FIG. 3, this NN 80 includes an input layer 90, a hidden layer 92 connected to receive an output of the input layer 90, and an output layer 94 connected to receive an output of the hidden layer 92.

  In this example, the input layer 90 includes three nodes. Each node is called a neural cell element (neuron). Each neuron outputs a value of a predetermined activation function whose input is the weighted sum of its inputs. The activation function may be a monotonically increasing function, but the activation function used in the present embodiment will be described later.

  Hidden layer 92 includes two hidden layer units 100 and 102. The hidden layer units 100 and 102 both have the same structure. Therefore, only the hidden layer unit 100 will be described below as an example. In addition, the number of hidden layers 92 is only one. Although two hidden layer units are shown in FIG. 3 for the purpose of explanation, the number of hidden layer units is about 10 in the present embodiment.

  The hidden layer unit 100 has COS elements 120, 122, 124, 126 and 128 each having an input coupled to the outputs of three input elements of the input layer 90 and using a cosine function as an activation function, and a COS element 120. , 、 Element 140 coupled thereto to receive the outputs of 122, 124, 126 and 128 and using an identity map as an activation function. Each of these elements is a type of neural cell element.

  Output layer 94 includes nodes coupled thereto to receive the outputs of hidden layer units 100 and 102, both in this embodiment.

  Weights and biases are set for the coupling between each element. The output of each Σ element is expressed by the following equation.

上式のベクトルｘは入力ベクトルを表し、行列ａ_ｄは入力層９０及び隠れ層ユニット１００、１０２の間の素子間の結合の重み行列を表し、ベクトルｂ_ｄは隠れ層ユニット１００及び１０２の各素子の出力に加算されるバイアスベクトルを表し、ベクトルｃ_ｄは隠れ層ユニット１００、１０２の各ＣＯＳ素子１２０、１２２、１２４、１２６及び１２８とΣ素子１４０との間の結合の重みベクトルを表す。これら重み及びバイアスを訓練データに基づいて決定する過程が学習である。学習の結果、各Σ素子は、入力層９０に対する入力について任意の連続関数を近似可能である。各Σ素子が近似する連続関数の周波数領域において、ＣＯＳ素子の入力側の重みは周波数成分に対応し、同入力側のバイアスは位相成分に対応し、同出力側の重みは振幅成分に対応する。各Σ素子は、互いにＣＯＳ素子の出力を共有しないことにより、入力空間上の任意の連続関数を独立に近似可能である。

The vector x in the above equation represents the input vector, the matrix a _d represents the weight matrix of the coupling between elements between the input layer 90 and the hidden layer units 100, 102, and the vector b _d represents each of the hidden layer units 100 and 102. represents a bias vector being added to the output of the device, the vector _{c d} represents the weight vector of the bond between each COS elements 120, 122, 124, 126 and 128 and Σ element 140 of the hidden layer units 100 and 102. The process of determining these weights and biases based on training data is learning. As a result of learning, each Σ element can approximate any continuous function for the input to the input layer 90. In the frequency domain of the continuous function approximated by each Σ element, the weight on the input side of the COS element corresponds to the frequency component, the bias on the input side corresponds to the phase component, and the weight on the output side corresponds to the amplitude component . Each Σ element can independently approximate any continuous function on the input space by not sharing the outputs of the COS elements with each other.

  The output layer 94 can be omitted by preparing the hidden layer units 100 and 102 as many as the number of labels. Further, by setting the activation function of the Σ element of each hidden layer unit as a softmax function, the output can be used as a probability value. The softmax function is a function (i = 1,..., n) expressed by the following equation.

  By preparing the output layer 94 as shown in FIG. 3, it is also possible to incorporate more or less Σ elements than the number of labels. The approximation performance of the network is equivalent even when a sine function is used instead of the cosine function in the COS element. Also, other differentiable periodic functions or combinations thereof may be used.

  Also in this embodiment, as in the conventional FFNN and the like, learning is performed using the error back propagation method. However, in the present embodiment, the cosine function is used as the activation function in the COS element of the hidden layer unit. Therefore, it is difficult to perform optimization learning of the hidden layer units 100 and 102 on the training data using the same method as the conventional method as described below. Therefore, in the present embodiment, the following two methods are used at the time of learning. The first is regularization, and the second is adoption of the quasi-Newton method.

  The network structure used in the present embodiment uses frequency domain representation of data distribution. Therefore, in the NN 80, it is possible to individually control each component of the frequency component, the phase component, and the amplitude component that can be interpreted as the property in the frequency domain of the approximating function. In general, the global tendency of a function is represented by low frequency components. Therefore, generalization performance can be improved by obtaining a global data distribution. Therefore, in the present embodiment, the data distribution is approximated by the low frequency region by performing regularization to frequency components. The regularization term is a downwardly convex function (convex function) with the origin at the top for parameters representing frequency components. Assuming that the objective function in learning is the error between the approximation function and the training data, adding the regularization to the error makes the objective function a minimum value near the origin and a convex function globally. FIGS. 4 and 5 show examples of the shape of the objective function regarding the value of the frequency parameter in the network used in the present embodiment. An example of the shape of the objective function without regularization is shown in FIG. Further, FIG. 5 shows an example of the shape of the objective function in the case where the L2 norm of the parameter representing the frequency component is added (L2 regularization) to the objective function to be minimized.

  When FIG. 4 and FIG. 5 are compared, it turns out that only one global minimum exists near the origin in FIG. 5 while FIG. 4 has a plurality of minimums. Therefore, each parameter is trained so that the error function converges to this global minimum. However, as is apparent from FIG. 5, there are a plurality of local minima (local minima) in this graph. In the normal learning process, there is a risk that the parameters may converge to one of these local minima. Such a situation must be avoided. Therefore, in the present embodiment, a global optimization method such as the quasi-Newton method including the L-BFGS method is further used for the frequency parameter. The quasi-Newton method is characterized in that it is difficult to converge to a local optimum solution. Among them, the L-BGFS method is known to use less memory. Therefore, it is possible to use an apparatus with few computational resources.

[Constitution]
FIG. 6 is a block diagram showing the character recognition device 180 and the character recognition NN learning device 186 for learning the NN of the character recognition device 180 as an example in which the NN having the above-described configuration is adopted.

  Referring to FIG. 6, character recognition device 180 receives input image 182 including handwritten characters, performs handwriting recognition, and outputs character recognition text 184. In this example, the character recognition device 180 receives an input image 182 from another scanner or storage device or the like (not shown). The character recognition device 180 receives the input image 182 and temporarily stores the input image storage device 200, and after correcting the inclination of the input image stored in the input image storage device 200, the individual character images Image recognition unit 202 for separating characters into character images and normalizing each character image, an image storage unit 204 for storing the character images for each character, and learned character recognition consisting of NN according to the present embodiment. It includes a NN 208 and a decoder 210 for reading out a character image for each character from the image storage unit 204, inputting the character image to the character recognition NN 208 and obtaining the result thereof to output a character recognition text 184.

  In the character recognition NN 208, learning by the character recognition NN learning device 186 is performed in advance. The character recognition NN learning device 186 uses a method similar to the image correction unit 202 based on a handwritten character database (DB) 240 that stores an image consisting of handwritten characters together with character labels of the characters, and handwritten characters stored in the handwritten character DB 240 A learning data generation unit 242 that corrects a character image and generates learning data in combination with character labels, a learning data storage unit 244 that stores learning data generated by the learning data generation unit 242, and a learning data storage unit 244 And a learning processing unit 246 for learning the character recognition NN 208 according to the above-described method using the learning data stored in the above. The number of nodes in the input layer of the character recognition NN 208 is equal to the number of elements in the learning data excluding the character label. The character label is used as teacher data at the time of learning to generate a correct vector used for error back propagation from the output layer side of the character recognition NN 208.

[Operation]
The character recognition NN learning device 186 and the character recognition device 180 described above operate as follows. First, learning of the character recognition NN 208 is performed by the character recognition NN learning device 186, and thereafter character recognition of the input image 182 by the character recognition device 180 is performed.

  In learning, the learning data generation unit 242 reads the handwritten character image stored in the handwritten character DB 240, corrects and normalizes the image as in the image correction unit 202, and combines with the character label to generate learning data. The generated learning data is accumulated in the learning data storage unit 244. The learning processing unit 246 uses the learning data stored in the learning data storage device 244 to learn the character recognition NN 208 by the method described above. When learning of the character recognition NN 208 by the learning processing unit 246 is finished, character recognition of the input image 182 by the character recognition device 180 becomes possible.

  In character recognition, an input image 182 is generated by a scanner (not shown) or the like and stored in the input image storage device 200. The image correction unit 202 corrects the image by performing inclination correction of the image, extraction of the character area, normalization of the character area, and the like, and causes the image storage unit 204 to store the corrected image. The decoder 210 reads this image from the image storage unit 204 and provides it to the character recognition NN 208 as an input. As a result of learning, the character recognition NN 208 returns to the decoder 210 an output that specifies a character label corresponding to the input character image. The decoder 210 generates and outputs the character recognition text 184 from the character label.

[Experimental result]
In order to investigate the performance of the above character recognition NN 208, experiments on handwriting character recognition were conducted. The experiments used the MNIST handwritten character data set. The performance according to the present embodiment and the maxout model shown in Non-Patent Document 6 as a discrimination model of NN and the performance based on rectifier shown in Non-Patent Document 5 are collectively shown in Table 1 below.

なお、表１において例えば「実施の形態（３０／１０）」の「３０」は隠れ層ユニット数を、「１０」は隠れ層ユニットのＣＯＳ素子数を、それぞれ表す。他も同様である。

In Table 1, for example, “30” in “Embodiment (30/10)” indicates the number of hidden layer units, and “10” indicates the number of COS elements in the hidden layer unit. Others are similar.

  As shown in Table 1, the identification performance of the NN according to the present embodiment does not reach that by the latest method. However, the performance is not too bad. Of particular note is the small number of parameters. In the embodiment (30/10), the number of parameters is 472K. On the other hand, at maxout, the number of parameters is 1233 K, and for rectifier, it is 3,798 K. As described above, according to the present embodiment, relatively high accuracy can be obtained with a small number of parameters. Furthermore, in the embodiment (20/10), the number of parameters is 157K, and in the embodiment (10/15), the number of parameters is further reduced to 118K. Nevertheless, the identification error rate increases only slightly, and the degradation in identification performance is negligible.

  Furthermore, in order to confirm how much regularization contributes to the improvement of the generalization performance, the case where the regularization is performed to learn the character recognition NN 208 and the case where the regularization is performed without the regularization are performed. The results of were compared. The case where regularization is performed is shown in FIG. 7, and the case where it is not performed is shown in FIG. In both figures, solid lines 260 and 280 indicate the identification error rate for the training data, and dashed lines 264 and 284 indicate the identification error rate for the test data. In the graph, the horizontal axis shows the number of repetitions of learning, and the vertical axis shows the discrimination error rate at that time.

  As shown by the solid line 280 in FIG. 8, in the case where regularization is not performed, identification errors for test data significantly decrease with learning. However, as indicated by the dashed line 284, the accuracy for the test data was barely high. On the other hand, as shown in FIG. 7, when regularization is performed, the decrease in error rate with respect to the learning data does not extend to the case where regularization is not performed as indicated by solid line 260. As shown by, the discrimination error rate for test data decreased with learning. That is, the identification error decreased and the identification accuracy increased. As a result, it can be said that the generalization performance is higher than in the case where regularization is not performed.

[Effect of the embodiment]
As described above, according to the above embodiment, generalization performance can be improved without the need for multi-layering. Therefore, the proportion of parallel calculation in the whole calculation increases, and the calculation efficiency when using a parallel computer is improved. In addition, the reduction in the number of layers reduces the latency in each layer, and reduces the calculation time at the time of prediction and the computer resources required for calculation.

  Since the Σ element, which is the calculation unit of the intermediate layer, does not share the outputs of the COS element as its input, the number of paths in error back propagation during training is reduced. In other words, the credit allocation problem is alleviated. Therefore, the NN according to the above-described embodiment can be optimized efficiently compared to the structure of full bonding, and has an effect of reducing training time.

  Since the conventional regularization aimed at reducing the number of parameters representing the distribution, the parameters involved in the identification remained part of the network. On the other hand, in the above embodiment, a periodically changing periodic function such as cosine or sine is used as the activation function. Therefore, all elements affect the identification result. As a result, it is possible to reduce the number of parameters to be prepared in advance, thereby reducing the overall calculation scale and enabling high-performance identification on a small-scale computer. In the above embodiment, a cosine function is used as the periodic function. However, as described above, a sine function may be used. A combination of a cosine function and a sine function may be used. Furthermore, a function obtained by piecewise linear approximation of a cosine function or a sine function (cosine function or sine function of piecewise linear approximation) is similarly a periodic function, and can be used instead of the cosine function or the sine function. When using a function that is piecewise linear approximated, training that applies the quasi-Newton method can not be performed by the usual method because differentiation is not possible in the connected part of line segments. However, even in such a case, if an inferior derivative can be defined (subdifferentiable) in a portion where differentiation can not be performed as in piecewise linear approximation, training can be performed by applying the quasi-Newton method. That is, the activation function used in the neural cell element of the hidden unit, such as the COS element, may be underdifferentiated.

  According to the above-mentioned embodiment, generalization performance can be obtained by smaller scale calculation than before. Therefore, when using the identification model in a mobile device with limited computing performance, it becomes possible to provide complete applications and services in the mobile device.

[Realization by computer]
Both the character recognition device 180 and the character recognition NN learning device 186 according to the embodiment of the present invention can be realized by computer hardware and a computer program executed on the computer hardware. FIG. 9 shows the appearance of the computer system 330, and FIG. 10 shows the internal configuration of the computer system 330.

  Referring to FIG. 9, computer system 330 includes a computer 340 having a memory port 352 and a DVD (Digital Versatile Disc) drive 350, a keyboard 346, a mouse 348 and a monitor 342.

  10, in addition to the memory port 352 and the DVD drive 350, the computer 340 includes a CPU (central processing unit) 356, a bus 366 connected to the CPU 356, the memory port 352 and the DVD drive 350, and a boot program. And a random access memory (RAM) 360 connected to the bus 366 for storing program instructions, system programs, work data and the like, and a hard disk 354. Computer system 330 further includes a network interface (I / F) 344 that provides a connection to network 368 that enables communication with other terminals.

  A computer program for causing the computer system 330 to function as the functional units of the character recognition device 180 and the character recognition NN learning device 186 according to the above-described embodiment is the DVD 362 or removable memory attached to the DVD drive 350 or the memory port 352 364 and further transferred to the hard disk 354. Alternatively, the program may be transmitted to the computer 340 via the network 368 and stored on the hard disk 354. The program is loaded into the RAM 360 upon execution. The program may be loaded from the DVD 362 into the RAM 360 directly from the removable memory 364 or via the network 368.

  This program includes an instruction sequence consisting of a plurality of instructions for causing the computer 340 to function as the functional units of the character recognition device 180 and the character recognition NN learning device 186 according to the above embodiment. Some of the basic functions necessary for causing the computer 340 to perform this operation are various operating tools or third party programs operating on the computer 340 or various dynamically linked programming toolkits or programs installed on the computer 340 Provided by the library. For example, software tools provided by commercially available statistical processing libraries can be used for the error back propagation method and the L-BFGS method. Therefore, the program itself does not necessarily include all the functions necessary to implement the system and method of this embodiment. This program dynamically calls at runtime the appropriate program in the appropriate function or programming toolkit or program library in a controlled manner in order to obtain the desired result of the instructions, the system or It only needs to include instructions that implement the function of the device. Of course, only the program may provide all the necessary functions.

[Application example]
The above embodiment relates to a handwritten character recognition apparatus. However, the present invention is not limited to such an embodiment. For example, it can also be used for acoustic models and statistical language models used in speech recognition devices. The present invention can also be applied to a translation model used in an automatic translation device and image pattern recognition other than handwritten characters.

  In the case of an acoustic model, not the speech itself but the feature quantities, for example, MFCC feature quantities obtained from speech signals, LPC feature quantities, mel filter bank outputs, their primary differences and their secondary differences, etc. The feature quantities that are usually used can be used as they are as input. The number of nodes in the input layer is made to coincide with the number of elements of the feature quantity vector. The number of nodes in the output layer is made to match the expected number of phonemes.

  In the case of the language model, for example, the following is performed. If it is a trigram language model, consider a vector consisting of all possible vocabulary. The first vector in which the element corresponding to the word two words earlier is 1 and the other element is 0, and the second vector in which the element corresponding to the word one word is 1 and the other elements are 0 And are connected together to form one feature vector. The number of nodes in the input layer is therefore twice the number of vocabulary expected. The number of nodes of the output node matches the expected number of vocabulary.

  Besides, the present invention is applicable to all machine learning and prediction devices using NN.

  The embodiment disclosed this time is merely an example, and the present invention is not limited to the above embodiment. The scope of the present invention is defined by each claim of the claims in consideration of the description of the detailed description of the invention, and all the changes within the meaning and range equivalent to the words and phrases described therein Including.

180 character recognition unit 182 input image 184 character recognition text 186 character recognition NN learning device 208 character recognition NN
210 decoder

Claims (5)

An input layer with multiple input nodes,
And a non-linear function layer having an input connected to receive an output from the input layer, the neural network representing a frequency domain in discrete Fourier transform of a continuous function,
The non-linear function layer includes a plurality of hidden layers each capable of approximating any independent component, each connected to receive an output from the plurality of input nodes of the input layer,
Each of the plurality of hidden layers is
Each of the plurality of input nodes has a weighted input representing a frequency in the frequency domain of the continuous function, and a bias representing a phase in the frequency domain of the continuous function as an activation function. A plurality of neural cell elements using a differentiable periodic function;
It is connected to receive the outputs of the plurality of neural cell elements in the same hidden layer with a weight representing the amplitude in the frequency domain of the continuous function, and the output aggregation using the underdifferentiable function as the activation function and a device, a neural network,
Further, an output layer connected to receive an output of the output aggregation element of the plurality of hidden layers,
The neural network, wherein the output layer comprises a plurality of output neural cell elements each connected to receive an output of the output aggregation element of the plurality of hidden layers in a weighted manner . The neural network according to claim 1, wherein the periodic function is a cosine function or a sine function or a combination thereof, or a piecewise linear approximated cosine function or a sine function. The neural network according to claim 1 or 2 trained using L2 regularization to an objective function using a weight parameter on the input side of each of the plurality of hidden layers. The neural network according to any one of claims 1 to 3, which is trained using a quasi-Newton method. A computer program that causes a computer to function as the neural network according to any one of claims 1 to 4 .

Images