CN107153873A - A kind of two-value convolutional neural networks processor and its application method - Google Patents

Abstract

The present invention provides a binary convolutional neural network processor, comprising: a storage device for data to be calculated, used to store elements of data to be convoluted in binary form and convolution kernel elements in binary form; a binary convolution device , for performing a binary convolution operation on the convolution kernel element in the binary form and corresponding elements in the data to be convoluted in the binary form; the data scheduling device is used for combining the convolution kernel element with the The corresponding elements in the data to be convoluted are loaded into the binary convolution device; the pooling device is used to perform pooling processing on the results obtained by convolution; and the normalization device is used to perform pooling processing on The results are normalized.

Description

A kind of binary convolutional neural network processor and using method thereof

technical field

The invention relates to the storage and scheduling of data used in neural network model calculation.

Background technique

With the development of artificial intelligence technology, technologies involving deep neural networks, especially convolutional neural networks, have developed rapidly in recent years. In image recognition, speech recognition, natural language understanding, weather prediction, gene expression, content recommendation It has been widely used in fields such as artificial intelligence and intelligent robots.

The deep neural network can be understood as an operation model, which contains a large number of data nodes, each data node is connected to other data nodes, and the connection relationship between each node is represented by weight. As deep neural networks continue to develop, so do their complexity.

In order to balance the contradiction between complexity and operation effect, in the reference: Courbariaux M, Hubara I, Soudry D, et al. Binarized neural networks: Training deep neural networks with weights and activations constrained to+1or-1[J].arXiv Preprint arXiv:1602.02830, 2016. It is proposed that the "binary convolutional neural network model" can be used to reduce the complexity of traditional neural networks. In the binary convolutional neural network, the weights, input data, and output data in the convolutional neural network are all in "binary form", that is, their size is approximately represented by "1" and "-1", such as "1" is used to indicate a value greater than or equal to 0, and "-1" is used to indicate a value less than 0. Through the above method, the data bit width used for operation in the neural network is reduced, thereby greatly reducing the required parameter capacity, making the binary convolutional neural network especially suitable for image recognition and augmented reality at the object end. and virtual reality.

In the prior art, a general-purpose computer processor is usually used to run a deep neural network, such as a central processing unit (CPU) and a graphics processing unit (GPU). However, dedicated processors for binary convolutional neural networks do not exist. The bit width of the calculation unit of a general-purpose computer processor is usually multi-bit, and the calculation of a binary neural network will result in waste of resources.

Contents of the invention

Therefore, the object of the present invention is to overcome the defective of above-mentioned prior art, a kind of binary convolutional neural network processor is provided, comprising:

The data storage device to be calculated is used to store the elements of the data to be convoluted in binary form and the convolution kernel elements in binary form;

A binary convolution device, configured to perform a binary convolution operation on the binary convolution kernel elements and corresponding elements in the binary data to be convoluted;

A data scheduling device, configured to load the convolution kernel element and the corresponding element in the data to be convolved into the binary convolution device;

a pooling device for pooling the results obtained by the convolution; and

A normalization device, used for normalizing the pooled results.

Preferably, according to the binary convolutional neural network processor, wherein the binary convolution device includes:

XNOR gate, which takes the convolution kernel element in the binary form and the corresponding element in the data to be convoluted in the binary form as its input;

an accumulating device, which takes the output of the XNOR gate as its input, and is used to accumulate the output of the XNOR gate to output the result of the binary convolution operation;

Wherein, the accumulating means comprises an OR gate and or a Hamming weight calculation unit, wherein,

At least one input of the OR gate is the output of the XNOR gate;

At least one input of the Hamming weight calculation unit is the output of the XNOR gate.

Preferably, according to the binary convolutional neural network processor, wherein the storage device for data to be calculated is also used to store the obtained binary-converted convolution kernel and or data to be convolved online .

Preferably, according to the binary convolutional neural network processor, it also includes:

The binarization device is used for converting the obtained convolution kernel and or the data to be convolved into a binary form.

Preferably, according to the binary convolutional neural network processor, a register is set in the data scheduling device for loading convolution kernel elements that need to be reused during use.

Preferably, according to the binary convolutional neural network processor described in any one of the above, the elements of the data to be convolved and the elements of the convolution kernel in the storage device for the data to be calculated are interleaved in layers And storage.

Preferably, according to the binary convolutional neural network processor, the elements of the data to be convoluted in the data storage device to be calculated are sequentially involved in the calculation according to the size of the convolution kernel and the convolution operation. The elements of the product data are stored.

Preferably, according to the binary convolutional neural network processor, the elements of the data to be convolved and or the elements of the convolution kernel in the storage device for the data to be calculated are stored in such a way that one or more of the following item:

Stored according to the matrix arrangement order of the convolution kernel and the data to be convolved;

Elements in the same position and in different channels in the matrix of the convolution kernel and or the data to be convoluted are continuously stored in multiple consecutive storage units;

All the elements under the same weight in the same convolution kernel and or all the elements in the sub-matrix used for the convolution operation in the same data to be convolved are stored in multiple consecutive storage units in the storage device.

Moreover, the present invention also provides a method for using the binary convolutional neural network processor described in any one of the above, including:

1) loading the data to be convoluted in the data storage device to be calculated into a register;

2) Load the data to be convolved in the register and the elements in the data to be calculated storage device that need to be multiplied with the data to be convolved into the binary convolution device to perform binary convolution accumulation operation;

3) performing pooling processing on the output of the binary convolution device by the pooling device;

4) performing a normalization operation on the output of the pooling device by the normalizing device.

And a computer-readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed.

Compared with the prior art, the present invention has the advantages of:

Provides a simplified hardware structure for performing convolution operations, a binary convolutional neural network processor based on this structure, and corresponding calculation methods. By reducing the bit width of the calculated data during the operation, the The effect of computing efficiency, reducing storage capacity and energy consumption.

Description of drawings

Embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

Fig. 1 is the schematic diagram of the multilayer structure of neural network;

Fig. 2 is a schematic diagram of convolution calculation in two-dimensional space;

FIG. 3 is a schematic diagram of the hardware structure of a binary convolution device according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of the hardware structure of a binary convolution device according to yet another embodiment of the present invention;

Fig. 5 is a schematic diagram of the hardware structure of a binary convolution device according to yet another embodiment of the present invention;

Figures 6a to 6c show a schematic diagram of the hardware structure of the binary convolution device using the Hamming weight calculation element in the present invention;

FIG. 7 is a schematic diagram of storing multi-channel convolution kernels, namely weight 0 and weight 1, and data to be convoluted according to an embodiment of the present invention;

8 is a schematic diagram of the structure of a binary convolutional neural network processor according to an embodiment of the present invention;

Fig. 9 is a schematic diagram of calculation using a binary convolutional neural network processor according to an embodiment of the present invention;

Fig. 10 is a schematic diagram of calculation using a binary convolutional neural network processor according to yet another embodiment of the present invention.

detailed description

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

The neural network in computer science is a mathematical model imitating the synaptic connection structure in biology. The application system composed of neural network can realize many functions such as machine learning and pattern recognition.

The neural network is structurally divided into multiple layers, and FIG. 1 shows a schematic diagram of a multi-layer structure of a neural network. Referring to Fig. 1, the first layer in the multi-layer structure is an input layer, the last layer is an output layer, and the remaining layers are hidden layers. When using the neural network, the original image is input to the input layer, that is, the input layer layer, (the "image" in the present invention, "layer" refers to the original data to be processed, not only in a narrow sense The image obtained by taking photos), each layer in the neural network processes the input layer and inputs the result to the next layer of the neural network, and finally uses the output of the output layer as the output result.

As mentioned above, in order to cope with the increasingly complex structure of the neural network, the prior art proposes a concept of a binary convolutional neural network. As the name implies, the operation of the binary convolutional neural network includes "convolution" operations on the input data, and it also includes operations such as "pooling", "normalization", and "binarization".

As an important operation in the binary convolutional neural network, . The calculation process of "convolution" will be introduced in detail through Figure 2 below.

FIG. 2 shows the calculation process of convolving a "binary" image with a size of 5x5 by using a "binary" convolution kernel with a size of 3x3 in a two-dimensional space. Referring to Figure 2, first, for each element in the 1-3 rows from top to bottom of the image, and the 1-3 columns from left to right, the corresponding elements in the convolution kernel and each of the elements are respectively used Multiplication: For example, use the element in row 1 and column 1 in the convolution kernel (expressed as "convolution kernel (1,1)") to multiply the element in row 1 and column 1 in the image (expressed as "image( 1,1)") to get 1×1=1, the convolution kernel (1,2) in row 1, column 2 in the convolution kernel is multiplied by the element image in row 1, column 2 in the image (1,2 ) to get 1×0=0, similarly calculate the convolution kernel (1,3) multiplied by the image (1,3) to get 1×1=1, and so on to get 9 results and add these 9 results Obtain 1+0+1+0+1+0+0+0+1=4 as the element of row 1 and column 1 in the convolution result, the convolution result (1,1). Similarly, compute kernel(1,1) times image(1,2), kernel(1,2) times image(1,3), kernel(1,3) times image( 1,4), the convolution kernel (2,1) is multiplied by the image (2,2)..., and so on to calculate 1+0+0+1+0+0+0+1=3 as the convolution result ( 1,2). The convolution result matrix with a size of 3 by 3 as shown in FIG. 2 can be calculated by using the above method.

The obtained convolution result as shown in Fig. 2 is input into the binary convolutional neural network of the next layer through buffering and binarization processing.

The above example shows the operations of "multiplication" and "addition" or "accumulation and summation" included in the calculation process of convolution.

The inventor realized that based on the unique characteristics of the binary multiplication operation, the "multiplication" in the binary convolution operation can be replaced by the "exclusive or not" operation, that is, only one logic element XNOR gate can be used to complete the present invention. In the existing technology, the operation of "multiplication" must be completed by using a multiplier. It can be seen that the binary-based convolution process is simpler than the traditional convolution, and it does not need to perform complex multiplication operations such as "2×4". When performing "multiplication" operations, if the multiplication operation If any of the elements in the multiplication operation is "0", the result obtained is "0", and if all the elements of the multiplication operation are "1", the result obtained is "1".

The principle that the XNOR gate element can be used to replace the multiplier in the present invention will be described in detail below through a specific example.

When actually using binarized convolution, the non-binary value z in the image and the convolution kernel is first binarized, that is:

Among them, the value z that is greater than or equal to 0 is binarized to "1" to represent the symbol "1" used in the convolution operation in Figure 2, and the value z that is less than 0 is binarized to "-1" to represent Figure 2 The symbol "0" used in the convolution operation.

Perform "XOR" operation on the binarized image and the value of the convolution kernel, that is There are several situations:

input A input B Output F symbol -1 -1 1 1 -1 1 -1 0 1 -1 -1 0 1 1 1 1

It can be seen from the above truth table that when the "multiplication" operation is performed on the binarized value, the logic element XNOR gate for performing the "exclusive NOR" operation can be used instead of the multiplier. However, as is well known in the art, the complexity of a multiplier is much higher than one logic element XNOR gate.

Therefore, the inventor believes that the complexity of the devices used in the binary convolutional neural network processor can be greatly reduced by using the logic element XNOR gate to replace the multiplier in the traditional processor.

In addition, the inventor also realized that the unique characteristics of the binary-based addition operation make the "addition" in the above-mentioned binary convolution operation be replaced by an "or" operation, that is, the logical element OR gate can be used to replace the The adder used in the prior art. This is because the result of the "OR" operation on the output of the above-mentioned XNOR gate can be expressed as G=F ₁ +F ₂ ...+F _n , and finally output a single-bit result G, where F _k represents the first Outputs of k XNOR gates, n represents the total number of XNOR gates whose outputs are used as inputs to OR gates.

Based on the above analysis of the inventor, the present invention provides a binary convolution device that can be used in a binary convolutional neural network processor, which uses the characteristics of binary-based multiplication and addition operations to simplify the processor The configuration of the hardware used to perform the convolution operation, thereby increasing the speed of the convolution operation and reducing the overall energy consumption of the processor.

FIG. 3 shows the hardware structure of a binary convolution device according to an embodiment of the present invention. As shown in FIG. 3 , the binary convolution device includes 9 XNOR gates and 1 OR gate, and the outputs of all 9 XNOR gates are used as the input of the OR gate. When performing convolution operation, each XNOR gate calculates n ₁ ×w ₁ , n ₂ ×w ₂ ...n ₉ ×w ₉ to obtain output F ₁ ~ F ₉ ; OR gate uses F ₁ ~ F ₉ as its Input, output the first element G ₁ in the convolution result. Similarly, by using the same convolution kernel to perform calculations on other areas in the image, the size of other elements in the convolution result can be obtained, which will not be repeated here.

In the embodiment shown in FIG. 3 , multiple XNOR gates are used in parallel to perform multiplication calculations, which increases the rate of convolution calculations. However, it should be understood that the hardware structure of the binary convolution device may also be modified in the present invention, which will be illustrated below through several other embodiments.

Fig. 4 shows the hardware structure of a binary convolution device according to yet another embodiment of the present invention. As shown in Figure 4, the binary convolution device includes 1 XNOR gate, 1 OR gate, and a register, the register is used to store the output of the OR gate and its stored value is used as the OR gate One of the inputs of the OR gate, and the other input of the OR gate is the output of the XNOR gate. When performing the convolution operation, according to the progress of time, n ₁ and w ₁ , n ₂ and w ₂ , ... n ₉ and w ₉ are used as the input of the XNOR gate at the first to ninth time respectively, corresponding to F ₁ , F ₂ . . . F ₉ are output from the XNOR gate at each moment as one input of the OR gate, and the result output from the OR gate at the previous moment stored in the register is used as the other input of the OR gate. For example, when the XNOR gate outputs F ₁ (its size is equal to n ₁ ×w ₁ ), read the pre-stored symbol "0" from the register and use it together with F1 as the input of the OR gate, and output F from the OR gate ₁ ; when the XNOR gate outputs F ₂ (its size is equal to n ₂ ×w ₂ ), read F ₁ from the register and use it and F ₂ together as the input of the OR gate, and output F ₁ +F from the OR gate ₂ , and so on until the accumulation result G ₁ for F ₁ to F ₉ is output.

In the embodiment shown in Figure 4, the number of components used is reduced by increasing the multiplexing rate of the XNOR gate and the OR gate, and the solution uses an OR gate with only two input terminals, and its hardware is complex to a lesser degree.

Fig. 5 shows the hardware structure of a binary convolution device according to yet another embodiment of the present invention. This embodiment is similar to the embodiment shown in Figure 4, all only adopting an XNOR gate, an OR gate and a register, the difference is that in Figure 5 the input of the XNOR gate is stored in a multi-bit result that can be stored simultaneously register, and the individual results in the register are used as inputs to the OR gate. The usage method of this embodiment is similar to the embodiment in Fig. 4, both are to multiplex the XNOR gate, the difference is that Fig. 5 stores the result output by the XNOR gate at each moment into a register capable of simultaneously storing multi-bit results , and after obtaining all F ₁ -F ₉ , the OR gate performs an "or" operation to output G ₁ .

In the embodiments provided in Fig. 3, 4, and 5 of the present invention, the OR gate is used to realize the function of "addition" or "accumulation", and the input of the OR gate comes from the output of the XNOR gate, so that the final The results output from the OR gate are all single-bit values, which can simplify the calculation process and increase the calculation rate. The hardware structure provided by this scheme is especially suitable for special-purpose processors for binary neural networks. This is because binary neural networks use the values "1" and "-1" to represent the weights and data in the neural network. There are a large number of multiplication and addition operations in the process, and reducing the bit width of the calculation operand can effectively reduce the calculation complexity.

However, since the above-mentioned schemes of using the OR gate to realize the function of "addition" or "accumulation" are all single-bit calculations, a certain degree of error will be introduced. In this regard, the present invention also provides an optional solution, that is, the Hamming weight calculation element is used to replace the OR gate shown in Figures 3, 4, and 5 to realize the "addition" or "accumulation" function. Fig. 6 a ~ 6c have shown the hardware structure that has the Hamming weight calculation element, in described alternative scheme, the Hamming weight calculation element takes the output of the XNOR gate as its input, outputs the logic "1" in the output data data, the Hamming weight. The solution described above is similar to the above-mentioned solution using the OR gate, and can also achieve the effect of simplifying the calculation process, and the solution can also achieve precise summing operations.

The inventors found that, based on the above-mentioned binary convolution device provided by the present invention, for each "multiplication" and "accumulation" calculation, the operation is a single bit of data, and the output of the binary convolution device is also All are single-bit data, and this feature is especially suitable for using the "layer interleaved data mapping method" to store and schedule the data obtained by participating in convolution operations and calculations, so as to reduce the number of data loading and make full use of data. The locality improves the effect of data reuse.

The "layer interleaved data mapping method" in the present invention refers to sequentially storing each element in the convolution kernel and the data to be convoluted into each row of the storage device according to the direction of the channel, that is In the storage device, data is stored in an interleaved manner, and two adjacent data elements come from different channels rather than the same channel. As shown in Figure 7, in the present invention, the convolution kernel on the same z-axis and the elements of the data to be convoluted correspond to the same "channel", that is, elements with the same z value belong to the same channel.

In order to describe the data calculation method more vividly and concretely, Fig. 7 uses (x, y, z) = 2*2*2 convolution kernel weight 0 and convolution kernel weight 1, and (x, y, z) = Taking 2*3*2 data to be convolved as an example, the layer interleaved data mapping method suitable for binary convolutional neural networks provided by the present invention is described in detail. Referring to Figure 7, the elements in weight 0 and weight 1 are divided into four groups according to the spatial position of the element: among them, the four groups of weight 0 are A _z , B _z , C _z and D _z , As shown in the figure, z is 0, 1; the four groups of weights of weight 1 are a _z , b _z , c _z and d _z respectively, and z is 0, 1 as shown in the figure.

Referring to FIG. 7 , according to an embodiment of the present invention, the convolution kernel weight 0, the convolution kernel weight 1, and each element in the data to be convolved can be stored in the following manner.

In Figure 7, for the convenience of illustration, according to the size and step size of each convolution kernel, the elements in the three-dimensional matrix with weight 0 and weight 1 are divided into two two-dimensional matrices according to the channel they are in, for example, weight 0 Divided into a two-dimensional matrix composed of A ₀ , B ₀ , C ₀ , D ₀ and a two-dimensional matrix composed of A ₁ , B ₁ , C ₁ , D ₁ ; similarly, the three-dimensional matrix of the data to be convoluted The elements in the matrix are divided into two two-dimensional matrices according to the channel they are in, that is, the two-dimensional matrix composed of X ₀ , Y ₀ , Z ₀ , P ₀ , Q ₀ , R ₀ and the two-dimensional matrix composed of X ₁ , Y ₁ , A two-dimensional matrix composed of Z ₁ , P ₁ , Q ₁ , and R ₁ .

When storing convolution kernel weight 0, elements A ₀ , A ₁ , B ₀ , B ₁ , C ₀ , C ₁ , D ₀ and D in weight 0 are sequentially stored in a row of continuous storage units of the weight storage device ₁ , 8 bits in total. It can be seen that in the storage unit, two adjacent elements come from different channels. For example, A ₀ and A ₁ come from different channels, and A ₁ and B ₀ also come from different channels. In this way, It is the interleaved storage method according to the layers mentioned above.

When storing convolution kernel weight 1, in another row of continuous storage units of the weight storage device, a ₀ , a ₁ , b ₀ , b ₁ , c ₀ , c ₁ , d of elements in weight 1 are sequentially stored ₀ and d ₁ , a total of 8 bits. Similar to the storage method of weight 0, two adjacent elements also come from different channels.

In the weight storage device, the weight elements on the same x-axis and the same y-axis (for example, A ₀ and A ₁ ) are stored as adjacent elements in sequence, and the next group with Weight elements of the same x-axis and y-axis (such as B ₀ and B ₁ ), and so on, store other weight elements in the convolution kernel.

When storing the data to be convoluted, it can be stored according to the size of the convolution kernel and the data elements sequentially involved in the calculation during the convolution operation. Referring to the convolution calculation rules shown in Figure 2, it can be seen that A _z X _z , B _z Y _z , C _z P _z and D _z Q _z need to be calculated first, and then A _z Y _z , B _z Z _z , C _z Q _z and D _z R _z for calculation. Therefore, when storing each element of the data to be convolved, in addition to the interleaved storage method of layers, the rules of convolution calculation should also be considered, so as to store the data elements involved in the calculation in sequence, such as X _z , Y _z , P _z , Q _z are stored in one row or one column of continuous memory cells, and Y _z , Z _z , Q _z , R _z are stored in another row or one column of continuous memory cells.

Referring to FIG. 7 , X ₀ , X ₁ , Y ₀ , Y ₁ , P ₀ , P ₁ , Q ₀ , and Q ₁ are sequentially stored in a column of continuous memory cells of the data storage device. In another column of continuous memory cells of the data storage device, Y ₀ , Y ₁ , Z ₀ , Z ₁ , Q ₀ , Q ₁ , R ₀ , and R ₁ are stored sequentially.

Similar to storing the elements of the convolution kernel, in the data storage device, the data elements (such as X ₀ and X ₁ ) located on the same x-axis and the same y-axis are grouped into one group and stored as adjacent elements in sequence, in After storing the elements of the same x-axis and the same y-axis, store the next set of weight elements with the same x-axis and y-axis (for example, Y ₀ and Y ₁ ), and so on, combine the data matrix to be convolved with the size of the convolution kernel Other data elements in the corresponding sub-matrix (eg, marked with dotted lines in FIG. 7) are stored.

Although in the example shown in FIG. 7 , the number of channels of the convolution kernel and the data to be convoluted is both 2, it should be understood that in the present invention, the convolution kernel and the data to be convoluted with the number of channels greater than 2 can also be calculated according to How layers are stored interleaved.

Preferably, when storing, a plurality of continuous storage units in the storage device are sequentially filled up, that is, stored in the storage device according to the matrix arrangement order of the convolution kernel and the data to be convolved.

Preferably, the convolution kernel and or the elements in the matrix of the data to be convolved at the same position but in different channels are continuously stored in a plurality of continuous storage units in the storage device.

Preferably, all elements under the same weight in the same convolution kernel and or all elements in the sub-matrix used for convolution operation in the same data to be convoluted are stored in a plurality of continuous storage units in the storage device.

In Fig. 7, for the convenience of explanation, the weight storage device and the data storage device are set as different storage devices from each other, but it should be understood that the present invention can respectively set the weight storage device and the data storage device on different memories, It can also be stored in different areas of the same memory, for example, stored collectively on the data storage device to be calculated.

Moreover, those skilled in the art should understand that the storage method described in the above embodiments can be completed offline outside the processor or online on the processor, for example, in the processing implemented in an on-chip chip of a device, or stored in the form of a computer program, and the computer program is executed by a processor.

Using the layer interleaved data mapping method according to the present invention to store each convolution kernel and each element in the data to be convoluted can reduce the number of data loading and improve the data multiplexing rate.

It should also be understood that the purpose of using the above-mentioned "layer interleaved data mapping method" to store the convolution kernel elements and the corresponding elements in the data to be convoluted is to facilitate reading, so as to quickly and conveniently determine the binary convolution device input of. Therefore, any manner that can establish a mapping relationship between the storage location of the convolution kernel element and the storage location of the corresponding element in the data to be convolved can be used to store the convolution kernel element and with the elements of the data to be convolved.

For example, when the length of the continuous storage unit is less than 8 bits, for example, only 4 bits, A ₀ , A ₁ , B ₀ , B ₁ , C ₀ , C ₁ , D ₀ and D ₁ in weight 0 are folded storage in the same way, that is, store A ₀ , A ₁ , B ₀ , and B ₁ in consecutive memory cells, and store C ₀ , C ₁ , D ₀ , and D ₁ in another row of consecutive memory cells.

When using the convolution kernel elements stored in the above manner and the corresponding elements in the data to be convoluted to perform convolution operations, it is suitable to use the single instruction multiple data flow (SIMD) method to perform, that is, the stored A plurality of data of is loaded into the operation unit. The method for loading and calculating the stored data will be described in detail in the following embodiments. In this way, the bit width of the computing unit can be reduced, and the hardware overhead of the computing unit can be reduced.

Combining the binary convolution device mentioned above and the storage and calling methods of the convolution kernel and the elements in the data to be convoluted, it is possible to provide a computing unit with a small amount of money and a relatively simple hardware structure for binary convolution A dedicated processor for neural networks.

Referring to FIG. 8, according to an embodiment of the present invention, a binary convolutional neural network processor 10 is provided, including:

A data scheduling device 101 , a storage device for data to be calculated 102 , a binary convolution device 103 , a pooling device 104 , a normalization device 105 , and a binarization device 106 .

Wherein, the storage device 102 for data to be calculated is used for storing convolution kernel elements in binary form and data to be convolved in binary form. As mentioned above, the storage method should be able to reflect the mapping relationship between the elements of the convolution kernel used for convolution calculation and the corresponding elements in the data to be convolved. For example, the elements of the convolution kernel and the data to be convolved are stored in an interleaved manner according to the layers, and the data to be convolved is stored according to the size of the convolution kernel and the elements of the data to be convolved that are sequentially involved in the calculation during the convolution operation. For a specific storage manner, reference may be made to the foregoing embodiments.

The data scheduling device 101 is configured to load the convolution kernel element and the corresponding element in the data to be convolved into the binary convolution device according to the mapping relationship. For example, registers are set in the data scheduling device 101, and the convolution kernel elements to be reused are loaded into the registers during use.

The binary convolution device 103 is configured to perform a binary convolution operation on the binary convolution kernel elements and corresponding elements in the binary data to be convoluted. The binary convolution device 103 can adopt any structure as in the foregoing embodiments, and realize the multiplication operation of the convolution kernel element and the corresponding element in the data to be convoluted through the XNOR gate, and through the OR gate or Hamming The weight calculation element implements the accumulation of the results obtained through the multiplication operation.

The pooling device 104 is configured to perform pooling processing on the result obtained by the convolution.

A normalization device 105 is configured to perform a normalization operation on the pooled results to speed up the parameter training process of the neural network.

In some embodiments of the present invention, the convolution kernel used for the binary convolution operation and or the data to be convolved can be obtained online from the data source. Since the obtained data is not necessarily binarized data, in the embodiment, a binarization device 106 can also be set in the binary convolutional neural network processor 10 to convert the obtained data into binary form. In addition, the binary-converted data can also be stored online by the data-to-be-calculated storage device 102 .

It should be understood that, for the embodiment in which the convolution kernel and or the data to be convolved have been stored offline in advance in the data storage device 102 before performing the convolutional neural network calculation, it is not necessary to A binarization device 106 is set in.

Referring to FIG. 9 and FIG. 10 , the calculation process using the binary convolutional neural network processor 10 shown in FIG. 8 will be described in detail through specific embodiments.

FIG. 9 shows a calculation process using the above-mentioned binary convolutional neural network processor according to an embodiment of the present invention. FIG. 9 uses the same symbols as those in FIG. 7 to represent convolution kernel elements and data elements to be convolved, for example, X ₀ , X ₁ , A ₀ , A ₁ and so on. In the weight storage matrix, all the convolution kernel elements in a row in the same channel are stored in one storage word. As shown in the figure, the bit width of the storage word is 8 bits, and each element occupies 1 bit. Similarly, the bit width of a storage word in the data matrix to be convoluted is also 8 bits. In addition, in FIG. 9, the bit widths of the XNOR gate and the register bank are both 2 bits. During the calculation, follow the principle that the data in the same convolution kernel is accumulated in the same accumulator. Its calculation process is as follows:

Step 1, load the upper two bits (namely X ₀ and X ₁ ) in the data to be convolved into the register set;

Referring to the convolution principle diagram shown in Figure 2, it can be seen that in Figure 9, the elements in the data to be convoluted will be repeatedly used X ₀ and X ₁ to calculate A ₀ X ₀ , B ₀ X ₀ , A ₁ X ₁ , B ₁ X ₁ , so it is necessary to store 2-bit data in the data to be convoluted into the register.

Step 2, load the data to be convolved in the register set and the first two weight data (A ₀ and A ₁ ) of the first row in the weight matrix into the XNOR gate;

Step 3, perform an OR operation or calculate the Hamming weight on the calculation result of the XNOR gate through the addition unit;

As mentioned above, the OR operation or the calculation of the Hamming weight can achieve the effect of "addition". In this step, A ₀ X ₀ and A ₁ X ₁ can be calculated.

Step 4, input the calculation result of the addition unit into the value accumulator 0;

The accumulator 0 is for accumulating data in the same convolution kernel.

Step 5, load the data to be convolved in the register set and the first two weight data (a ₀ and a ₁ ) of the second row in the weight matrix into the XNOR gate;

In step 6, the adding unit performs an OR operation on the calculation result of the XNOR gate or calculates the Hamming weight to obtain a ₀ X ₀ and a ₁ X ₁ .

Step 7, input the calculation result of the addition unit into the accumulator 1, and so on, calculate the weights of X ₀ and X ₁ with the first two digits of the specified eight rows in the weight storage array in turn;

Step 8, similar to the previous steps, load the third and fourth bits (Y ₀ and Y ₁ ) in the data to be convolved into the register set;

Step 9, loading the data to be convolved in the register set and the third and fourth weight data (B ₀ and B ₁ ) of the first row in the weight matrix into the XNOR gate;

Step 10, performing an OR operation or calculating the Hamming weight on the calculation result of the XNOR gate through the addition unit;

Step 11, input the calculation result of the addition unit into the value accumulator 1, and thereafter, similar to steps 5 to 7, calculate b ₀ and b ₁ in the same column with Y ₀ and Y ₁ in turn;

Step 12, when the accumulator obtains the data of an output layer, load the calculation result of the accumulator into the buffer unit;

Step 13, after the buffer unit obtains the complete data of the output layer, load the output data to be convoluted into the pooling unit for pooling operation;

Step 14, load the calculation result of the pooling operation into the batch normalization unit to perform the batch normalization operation;

Step 15, load the calculation result of batch normalization into the binarization unit to perform binarization operation.

It can be seen that, according to the mapping relationship between the storage location of the convolution kernel element and the storage location of the corresponding element in the data to be convoluted, it can be quickly determined that convolution corresponding element of the product to feed it into the XNOR gate.

When the bit width of the storage unit is smaller than the bit width of the matrix shown in FIG. 9 , the matrix can also be folded in blocks to store convolution kernel elements and data elements to be convolved, as shown in FIG. 10 . Similarly, Figure 10 also uses the same symbols as those in Figure 7 to describe the convolution kernel elements and data elements to be convoluted, the difference from Figure 9 is that when it is necessary to read the data in the same data block that belongs to the data to be convolved You also need to consider where the data is stored in the register file.

It can be seen from the embodiments of the present invention that the present invention provides a simplified hardware structure for performing convolution operations based on the characteristics of binarization operations, and a binary convolutional neural network processor based on the structure and corresponding By reducing the bit width of the calculated data during the calculation process, the calculation method can improve the calculation efficiency and reduce the storage capacity and energy consumption.

Moreover, the present invention adopts a layer interleaved data mapping method for data storage and calculation, which simplifies the process of calling data to be convoluted and convolution kernel data during convolution calculation, reduces hardware overhead and improves data utilization.

It should be noted that not all the steps described in the foregoing embodiments are necessary, and those skilled in the art may make appropriate trade-offs, replacements, modifications, etc. according to actual needs.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than limit them. Although the present invention has been described in detail above with reference to the embodiments, those skilled in the art should understand that modifications or equivalent replacements to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all of them should be covered in Within the scope of the claims of the present invention.

Claims (10)

1. A binary convolutional neural network processor, comprising: The data storage device to be calculated is used to store the elements of the data to be convoluted in binary form and the convolution kernel elements in binary form; A binary convolution device, configured to perform a binary convolution operation on the binary convolution kernel elements and corresponding elements in the binary data to be convoluted; A data scheduling device, configured to load the convolution kernel element and the corresponding element in the data to be convolved into the binary convolution device; a pooling device for pooling the results obtained by the convolution; and A normalization device, used for normalizing the pooled results. 2. The binary convolutional neural network processor according to claim 1, wherein said binary convolution device comprises: XNOR gate, which takes the convolution kernel element in the binary form and the corresponding element in the data to be convoluted in the binary form as its input; an accumulating device, which takes the output of the XNOR gate as its input, and is used to accumulate the output of the XNOR gate to output the result of the binary convolution operation; Wherein, the accumulating means comprises an OR gate and or a Hamming weight calculation unit, wherein, At least one input of the OR gate is the output of the XNOR gate; At least one input of the Hamming weight calculation unit is the output of the XNOR gate. 3. The binary convolutional neural network processor according to claim 1, wherein the data storage device to be calculated is also used to convert the obtained convolution kernel and or the data to be convoluted through binary conversion on-line to store. 4. The binary convolutional neural network processor according to claim 3, further comprising: The binarization device is used for converting the obtained convolution kernel and or the data to be convolved into a binary form. 5. The binary convolutional neural network processor according to claim 1, wherein said data scheduling device is provided with registers for loading convolution kernel elements that need to be reused during use. 6. The binary convolutional neural network processor according to any one of claims 1-5, in the data storage device to be calculated, the elements of the data to be convoluted and the convolution kernel elements are according to the diagram stored in an interleaved manner. 7. The binary convolutional neural network processor according to claim 6, in the data storage device to be calculated, the elements of the data to be convolved are involved in the calculation according to the size of the convolution kernel and the convolution operation. The elements of the data to be convolved are stored. 8. The binary convolutional neural network processor according to claim 7, in the data storage device to be calculated, the elements of the data to be convolved and or the storage mode of the convolution kernel elements meet the following one or more: Stored according to the matrix arrangement order of the convolution kernel and the data to be convolved; Elements in the same position and in different channels in the matrix of the convolution kernel and or the data to be convoluted are continuously stored in multiple consecutive storage units; All the elements under the same weight in the same convolution kernel and or all the elements in the sub-matrix used for the convolution operation in the same data to be convolved are stored in multiple consecutive storage units in the storage device. 9. A method for using the binary convolutional neural network processor according to any one of claims 1-8, comprising: 1) loading the data to be convoluted in the data storage device to be calculated into a register; 2) Load the data to be convolved in the register and the elements in the data to be calculated storage device that need to be multiplied with the data to be convolved into the binary convolution device to perform binary convolution accumulation operation; 3) performing pooling processing on the output of the binary convolution device by the pooling device; 4) performing a normalization operation on the output of the pooling device by the normalizing device. 10. A computer-readable storage medium in which is stored a computer program for implementing the method as claimed in claim 9 when executed.