JP3262340B2 - Information processing device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information processing apparatus imitating a neural network.

[0002]

2. Description of the Related Art In recent years, information processing apparatuses imitating neural networks have been developed and have been actively used in pattern classification apparatuses such as character recognition apparatuses. Such an information processing apparatus is described in, for example, a document (PDP model, DE
The configuration described in Ramelhart and two others, translated by Shunichi Amari, 1989) is known.

Hereinafter, the configuration will be briefly described with reference to FIG. FIG. 4 shows a schematic configuration of an information processing apparatus simulating a neural network having a three-layer structure of a first layer X, a second layer Y, and a third layer Z. 1X, 1Y, and 1Z are signal conversion circuits that generally have a plurality of neural elements internally including a linear conversion unit and a non-linear conversion unit. 2X, 2Y and 2Z are correction circuits for rewriting the characteristics of the linear conversion units in the signal conversion circuits 1X, 1Y and 1Z. An error calculation circuit 3 calculates an error signal D (z) to be sent to the correction circuit 2Z from the ideal output T and the actual output S (z).

Next, the operation of the information processing apparatus will be described. In an information processing device imitating a neural network, a plurality of signals flow in parallel in most cases, but distinguishing all of them by subscripts merely complicates the expression. In the description, the suffix will be omitted unless it is particularly necessary.

In the operation of the information processing apparatus simulating a neural network at the time of learning, the signal conversion circuit 1X outputs the input signal S
(i) is converted into a signal S (x) and output.
Stores the signal S (i) and the signal S (x) inside and stores the error signal D
Wait for (x) input. Similar processing is performed by the signal conversion circuit 1
Y, 1Z and correction circuits 2Y, 2Z. Then, the final output S (z), the ideal output corresponding to the input signal S (i), and T, also called a teacher signal, are sent to the error calculation circuit 3. In the error calculation circuit 3, the output signal S
(z) and the teacher signal T, an error signal Dj (z) related to the j-th neural element of the third layer Z is calculated, for example, as follows:

Dj (z) =-μ (Sj (z) -Tj) Here, μ is calculated as a positive constant, and an error signal D (z) obtained by summing the values is obtained as a correction circuit 2
Sent to Z. The correction circuit 2Z outputs the held signal S
(y), S (z) and the error signal D (z), and the modified signal M (z)
Is sent to the signal conversion circuit 1Z, and the coupling strength Wji (z) of the linear conversion unit between the j-th neural element in the signal conversion circuit 1Z and the neural element in the signal conversion circuit 1Y is calculated as

Wji (z) + Dj (z) · Sj (z) · (1−Sj (z)) ·
Modified to Si (y), and as an error signal D (y), for example,

ΣDj (z) ・ Sj (z) ・ (1-Sj (z)) ・ Wji (z) are sent to the correction circuit 2Y. Hereinafter, similar processing is performed in the signal conversion circuits 1Y and 1X and the correction circuits 2Y and 2X. By repeating this procedure called learning,
An information processing apparatus imitating a neural network receives an input signal S
By learning the relationship between (i) and the ideal output T, an appropriate output can be output even for a new input inferred from past experience.

In this way, when the information processing device simulating a neural network performs sufficient learning and the error D (z) output from the error calculation circuit 3 becomes smaller than a certain value, the correction circuits 2Z, 2Y, The procedure related to 2X is omitted, and only the processing in the forward direction from the input to the output is performed.

Since such a learning method requires a long time for convergence, a device for speeding up the learning is disclosed in, for example, JP-A-1-320565. The learning efficiency improvement method shown therein prepares a set of several learning parameters (a parameter η for determining the step size and a parameter α for adjusting the amount of momentum), and proceeds with learning for each. The error in each case is calculated, and the learning parameter with the least error is selected.

[0011]

However, in the above-mentioned conventional learning efficiency improving method, since the error calculation for a plurality of learning parameters is performed at every set number of times even if the learning efficiency is improved, an enormous amount of calculation is required. There was a problem that it became necessary.

An object of the present invention is to solve such a conventional problem, and an object of the present invention is to provide an information processing apparatus simulating a neural network capable of performing more efficient learning.

[0013]

According to the present invention, there is provided an information processing apparatus simulating a neural network having a plurality of neural elements having a plurality of synapse loads. has a plurality learning mode in which, the
A plurality of learning modes are set every time one input signal is input.
Mode to modify the strength of the connection and the set multiple times
Correct coupling strength when only input signal is input
Mode, wherein the plurality of learning modes are switched according to the output error of the neural element.

[0014]

According to the present invention, the synapse load is corrected by the above configuration.
Correct multiple learning modes according to neural element output error
By switching the learning mode, a learning mode suitable for the initial stage of learning and a learning mode suitable for the latter period of the learning can be switched, and efficient learning can be performed.

[0015]

Embodiments of the present invention will be described below with reference to the accompanying drawings. FIG. 1 is a block diagram showing a schematic configuration of an information processing apparatus according to one embodiment of the present invention. The information processing apparatus includes a first layer X, a second layer Y, a third layer
The layer Z has a three-layer structure. 11X, 11Y, 11Z
Is a signal conversion circuit, and 12X, 12Y, and 12Z are correction circuits. 13 is an error calculation circuit, and 14 is a control circuit.

FIG. 2 is a block diagram showing a specific configuration example of the correction circuits 12X, 12Y and 12Z in the information processing apparatus shown in FIG. 121 is a first correction circuit, and 12
2 is a second correction circuit, and 123 is a switching circuit.

The operation of the above embodiment will be described below with reference to FIGS. In the present embodiment, the information processing device has an error calculation mode and a learning mode.

In the error calculation mode, the signal conversion circuits 11X,
In this mode, forward calculation is performed without changing the internal states of 11Y and 11Z, and an error is calculated. This mode is normally executed at the start of learning and at every set number of times, depending on the learning method.

The learning mode is a mode in which learning is performed in accordance with a learning method selected by a control signal from the control circuit 14. For example, when using the error backpropagation method, as a learning method at the initial stage of learning, a method of correcting the synapse load every time an input is received is advantageous. The method of correcting the synapse load at once after receiving the input as many times as can be statistically estimated is more advantageous. In the following description, a multi-layer perceptron having the above two learning algorithms will be described as an example, but the modification circuit 1 shown in FIG.
The first correction circuits 121 in 2X, 12Y, and 12Z operate according to the former learning algorithm, and
2 operates according to the latter learning algorithm.

First, at the start of learning, the control circuit 14 is reset, and the control circuit 14 corrects the correction circuits 12Z and 12Z.
A control signal for the error calculation mode is output to Y and 12X.
The calculation of the error is performed based on this, and the result is held in the control circuit 14 as an initial error.

Next, the control circuit 14 converts a control signal for selecting a learning method suitable for the initial stage of learning into correction circuits 12Z, 12Y,
The signal is sent to the switching circuit 123 in the 12X.
The correction circuit 121 is selected and learning is started. In addition,
In the configuration of the present embodiment, the correction circuit that does not operate outputs 0
And

The learning algorithm of the first correction circuit 121 selected by the switching circuit 123 is a method of correcting a synapse load every time an input is received, and performs the same operation as that described in the conventional example.

When the output error decreases at a fixed rate as the learning progresses, the control circuit 14 issues a control signal and operates the switching circuit 123. The switching circuit 123 stops the first correction circuit 121, operates the second correction circuit 122, and receives a plurality of inputs, preferably as many times as possible as many times as possible to statistically estimate the input distribution, and then corrects the synapse load at a time. The learning is advanced based on.

When the information processing device simulating the neural network performs sufficient learning and the error output from the error calculation circuit 13 becomes smaller than a predetermined value, the correction circuits 12X and 12
Procedures for Y and 12Z are omitted, and processing only in the forward direction from input to output is performed.

The control signal output from the control circuit 14 for switching the learning algorithm may be output when the amount of change in the connection weight becomes equal to or less than a set value.

In the first embodiment, the correction circuit 1
2 X, 12Y, and 12Z are the first correction circuit 121 and the second
It is composed of a correction circuit 122 and a switching circuit 123. If the learning algorithm used in the above embodiment is used, the correction circuits 12X, 12Y, and 12Z will
The configuration shown in FIG.

Next, a second embodiment of the present invention will be described with reference to FIG. In FIG. 3, reference numeral 124 denotes a variation relating to a synaptic load of an error for each input;

A variation calculation unit for calculating d E / d Wji | pattern, 125 is a counter 12
An integration unit that integrates the variation results until a signal from 6 is received. 127 is the last update amount that is reset when the mode is switched,

An update amount buffer unit for ΔWji old is provided, and 128 is a learning rate μ corresponding to the number of integrations in the integration variation output from the integration unit 125.
Multiply by

A learning rate conversion unit for outputting μ · Σd E / d Wji | pattern, and 129 multiplies the previous update amount by the momentum parameter α,

A momentum conversion section for outputting α · ΔWji old. Counter 12
6, an integration unit 125 and an update amount buffer unit 127 so that the number of integration is determined in response to a control signal from the control circuit 14, the output of the variation calculation unit 124 is counted, and the output is performed when the set number of times is reached. Send a signal to

The operation of the second embodiment will be described below. When a learning algorithm that modifies the synapse load for each input is used, the set value of the counter 126 becomes 1 by a control signal from the control circuit 14, and when the other learning algorithm is used, the input space is statistically calculated. The number N can be estimated.

In the course of the forward calculation of the information processing apparatus, the variation calculation section 124 has the signal conversion circuit 12X,
The output of 12Y is input. In the case of the backward calculation, the error signal from the error calculation circuit 13 is input to the variation calculation unit 124, and the variation regarding the synaptic load of the error for each input is calculated.

The d E / d Wji | pattern and the error signal to be sent to the next correction circuit 12Y or 12X are obtained, and are sent to the integrating section 125, the counter 126 and the next correction circuit 12Y or 12X, respectively. The counter 126 counts the number of signals sent from the variation calculation unit 124, and keeps counting until it becomes equal to a set value (1 or N) determined by a signal from the control circuit 14.
The integration unit 125 and the update amount buffer unit 127 are stopped so that no signal is output. When the count number becomes equal to the set value, the counter 126 outputs the integration result to the integration unit 125 and the update amount buffer unit 127,

Σd E / d Wji | pattern and the previous update amount,

.DELTA.Wji old is output to the learning rate conversion unit 128 and the momentum conversion unit 129, respectively, and both are set internally. The learning rate conversion section 128 multiplies the result of integration in the integration section 125 by a learning rate μ determined by a signal from the control circuit 14 and outputs the result. The momentum conversion unit 129 multiplies the previous update amount by the momentum parameter α and outputs the result. These two outputs are added and the update amount,

ΔWji = μ · Σd E / d Wji | pattern + α · ΔWji old and is sent to the signal conversion circuits 11X, 11Y and 11Z and the update amount buffer unit 127.

As described above, in the second embodiment, the switching between the two learning modes is realized by controlling the set value of the counter 126 and the magnitude of the learning rate by the control circuit 14.

In the description of the second embodiment, only the learning rate is changed when the learning mode is changed in order to avoid the complexity of the description. However, the momentum parameters can be changed simultaneously or independently.

Although the first and second embodiments have been described with reference to an example in which a multi-layer perceptron is trained by an error back propagation method, the present invention relates to a multi-layer perceptron as an information processing apparatus simulating a neural circuit. The learning algorithm is not limited to the backpropagation method. For example, the present invention can be applied to a learning vector quantization method or the like, and it goes without saying that any number of learning modes may be used.

Further, although the control signal is output when it becomes equal to or less than the set value, the control signal may be generated according to the range of the error.

[0042]

As described above, according to the present invention, a plurality of
Simulate neural network with neural elements with synaptic loads
In the information processing device, the synapse load is corrected.
Output modes of the neural element.
, The plurality of learning modes are switched according to the above , so that the learning mode suitable for the initial stage of learning and the learning mode suitable for the latter period of the learning can be switched, and more efficient learning can be performed.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a schematic configuration of a first embodiment of the present invention.

FIG. 2 is a block diagram showing a specific configuration example of a correction circuit according to the first embodiment of the present invention.

FIG. 3 is a block diagram showing a specific configuration example of a correction circuit according to a second embodiment of the present invention.

FIG. 4 is a block diagram showing a schematic configuration of an information processing apparatus imitating a conventional neural circuit.

[Explanation of symbols]

 11X, 11Y, 11Z signal conversion circuit 12X, 12Y, 12Z correction circuit 13 error calculation circuit 14 control circuit 121 first correction circuit 122 second correction circuit 123 switching circuit 124 variation calculation section 125 integration section 126 counter 127 update amount buffer section 128 Learning rate converter 129 Momentum converter

Continuation of the front page (56) References JP-A-2-77961 (JP, A) JP-A-2-277180 (JP, A) JP-A-1-271888 (JP, A) Takashi Morie et al. Circuit Study of Analog Neuro LSI with Built-in Propagation Learning Function ", IEICE Technical Report, Japan, The Institute of Electronics, Information and Communication Engineers, Japan, October 25, 1990, Vol. 90, No. 273 (CPSY90-64 to 74), pp. 43-48, (Patent Office CSDB literature number: C SNT199900846006) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06N 1/00-7/00 G06G 7/60 CSDB (Japan Patent Office) JICST file (JOIS)

Claims (5)

(57) [Claims] 1. A data processing apparatus simulating a neural network with neural element having a plurality of synaptic weights, having a plurality learning mode for modifying the synapse loads, the
A plurality of learning modes are set every time one input signal is input.
Mode to modify the strength of the connection and the set multiple times
Correct coupling strength when only input signal is input
Sequence by the mode, the information processing apparatus characterized by switching the plurality learning mode in response to the output error of the neural elements. 2. The information processing apparatus according to claim 1 , wherein the switching of the learning mode is performed when the output error becomes equal to or less than a set value. 3. The information processing apparatus according to claim 1 , wherein the switching of the learning mode is performed when the output error becomes equal to or less than a set ratio of the output error at the start of learning. Switching of 4. A learning mode, claim to be performed when a change amount of the connection weight is below the value set
The information processing apparatus according to claim 1 . 5. An information processing apparatus simulating a neural network having a plurality of neural elements for converting and outputting an input signal by using a synapse load, wherein the neural element has a plurality of learning modes for correcting the synapse load; An error calculation circuit that calculates an output error between the output signal of the neural element and the teacher signal,
And a control circuit for outputting a control signal for switching the plurality learning mode in response to the output error, said plurality
Learning mode is connected every time one input signal is input
Mode to correct the strength of the
A mode that modifies the strength of the connection when a force signal is input
And an information processing device simulating a neural network.