CN109754062A - Execution method of convolution expansion instruction and related products - Google Patents

Abstract

Present disclosure provides the implementation method and Related product of a kind of convolution extended instruction, comprising: computing device reads input data, convolution kernel and the auxiliary operation that the convolution extended instruction obtains the convolution extended instruction from memory；The convolution extended instruction includes: operation code and operation domain, and the operation domain includes: register and auxiliary domain, and the register is used to determine the address of input data and the address of convolution kernel, and the auxiliary domain is for identifying auxiliary operation；Computing device executes convolution operation and auxiliary operation to the address of the input data and the address of convolution kernel.The technical solution that present disclosure provides has the advantages of reducing calculation amount, reducing power consumption.

Description

Execution method of convolution expansion instruction and related products

technical field

The present disclosure relates to the technical field of neural networks, and in particular, to a method for implementing a convolution expansion instruction and related products.

Background technique

Convolutional neural network is an efficient recognition algorithm that has been widely used in pattern recognition, image processing and other fields in recent years. It has the characteristics of simple structure, few training parameters, strong adaptability, translation, rotation and scaling. Since the feature detection layer of CNN/DNN learns through training data, when using CNN/DNN, it avoids explicit feature extraction and implicitly learns from training data; The meta weights are the same, so the network can learn in parallel, which is also a big advantage of convolutional networks over networks where neurons are connected to each other.

Among the existing applications in the computer field, applications related to convolution operations are very common. The present invention focuses on convolutional neural networks, and currently the mainstream devices that can perform such operations are as follows:

In the prior art, a known solution for performing convolutional neural network operations is to use a general-purpose processor, which executes general-purpose instructions through a general-purpose register file and general-purpose functional components to perform convolutional neural network operations. However, one of the disadvantages of this method is that a single general-purpose processor is mostly used for scalar computation, and the computational performance is low when performing convolutional neural network operations. When using multiple general-purpose processors to execute in parallel, the mutual communication between the general-purpose processors may become a performance bottleneck.

In another prior art, a graphics processing unit (GPU) is used for vector computations, where convolutional neural network operations are performed by executing general-purpose SIMD instructions using a general-purpose register file and a general-purpose stream processing unit. However, in the above solution, the on-chip cache of the GPU is too small, which requires continuous off-chip data transfer when performing large-scale convolutional neural network operations, and the off-chip bandwidth becomes the main performance bottleneck.

Disclosure

Embodiments of the present disclosure provide a method for implementing a convolution expansion instruction, a convolution expansion instruction, and related products, which can achieve the advantages of improving performance bottlenecks and reducing power consumption.

In a first aspect, an embodiment of the present disclosure provides a method for executing a convolution expansion instruction, and the method includes the following steps:

The computing device reads the convolution expansion instruction from the memory to obtain the input data, convolution kernel and activation operation of the convolution expansion instruction;

The convolution extension instruction includes: an operation code and an operation field, the operation code includes: the identifier of the convolution extension instruction; the operation field includes: a convolution subfield and an activation subfield, and the convolution subfield Including: storing the address of the input data and the address of the convolution kernel, and the activation subfield includes: the identification code of the activation operation or the interpolation table address of the activation operation;

The computing device performs a convolution operation on the input data and the convolution kernel to obtain an intermediate result, and performs an activation operation on the intermediate result through the activation subfield to obtain a final result of the instruction.

Optionally, the activation operation includes: a convolutional neural network Maxout operation, a convolutional neural network PReLU operation, a convolutional neural network RReLU operation, a convolutional neural network Leaky ReLU operation, a nonlinear activation operation or a linear activation operation.

Optionally, if the activation subfield includes: an interpolation table address of the activation operation, the final result of the instruction is obtained by performing the activation operation on the intermediate result through the activation subfield, including:

The computing device extracts the interpolation table corresponding to the interpolation table address of the activation operation, and performs the activation operation on the intermediate result and the interpolation table to obtain the final result of the instruction.

Optionally, if the activation subfield includes: an identification code of the activation operation, the final result of the instruction is obtained by performing the activation operation on the intermediate result through the activation subfield, including:

The computing device identifies the identification code of the activation operation to determine the activation operation, reads the interpolation table of the activation operation, and executes the activation operation on the interpolation table and the intermediate result to obtain the final result of the instruction.

Optionally, the computing device performs a convolution operation on the input data and the convolution kernel to obtain an intermediate result, including:

The main operation module of the computing device splits the input data into multiple parts to obtain multiple input sub-data, distributes the multiple input sub-data to multiple slave operation modules, and sends the convolution kernel to multiple slave operation modules, The multiple slave operation modules execute the multiplication operation of the input sub-data and the convolution kernel in parallel to obtain multiple sub-results, and the main operation module of the computing device splices the multiple sub-results to obtain the intermediate result.

In a second aspect, a computing device is provided, the computing device comprising: a memory, an arithmetic unit, an interconnection module, an arithmetic unit, a controller unit, and a data access unit;

Wherein, the arithmetic unit includes: an adder and a multiplier;

a controller unit, configured to read the convolution expansion instruction from the memory to obtain the input data, convolution kernel and activation operation of the convolution expansion instruction;

The convolution extension instruction includes: an operation code and an operation field, the operation code includes: the identifier of the convolution extension instruction; the operation field includes: a convolution subfield and an activation subfield, and the convolution subfield Including: storing the address of the input data and the address of the convolution kernel, and the activation subfield includes: the identification code of the activation operation or the interpolation table address of the activation operation;

A data access unit, used to obtain the input data and the convolution kernel corresponding to the address of the input data and the address of the convolution kernel;

The operation unit is configured to perform a convolution operation on the input data and the convolution kernel to obtain an intermediate result, and perform an activation operation on the intermediate result through the activation subfield to obtain the final result of the instruction.

Optionally, the activation operation includes: a convolutional neural network Maxout operation, a convolutional neural network PReLU operation, a convolutional neural network RReLU operation, a convolutional neural network Leaky ReLU operation, a nonlinear activation operation or a linear activation operation.

Optionally, the activation subfield includes: the interpolation table address of the activation operation;

The data access unit is used to extract the interpolation table corresponding to the interpolation table address of the activation operation;

The operation unit is configured to perform an activation operation on the intermediate result and the interpolation table to obtain the final result of the instruction.

Optionally, the activation subfield includes: an identification code of the activation operation; the operation unit further includes: an activation operator;

the controller unit, for identifying the activation operation by identifying the identification code of the activation operation;

The activation calculator is configured to obtain the interpolation table of the activation operation, and perform the activation operation on the interpolation table and the intermediate result to obtain the final result of the instruction.

Optionally, the operation unit further includes: a master operation module and a plurality of slave operation modules, the master operation module includes: an adder and a multiplier, and the slave operation module includes: an adder and a multiplier ;

The main operation module is used to split the input data into multiple parts to obtain multiple input sub-data, distribute the multiple input sub-data to multiple slave operation modules, and send the convolution kernel to multiple slave operations module, the multiple slave operation modules are used for performing the multiplication operation of the input sub-data and the convolution kernel in parallel to obtain multiple sub-results, and the master operation module is used for splicing the multiple sub-results to obtain the intermediate result.

A third aspect provides a computer-readable storage medium, characterized in that it stores a computer program for electronic data exchange, wherein the computer program causes a computer to execute the method provided in the first aspect.

In a fourth aspect, a computer program product is provided, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform the method of the first aspect.

A fifth aspect provides a chip, where the chip includes the computing device provided in the second aspect.

In a sixth aspect, a chip packaging structure is provided, and the chip packaging structure includes the chip provided in the fifth aspect.

According to a seventh aspect, a board card is provided, and the board card includes the chip packaging structure provided in the sixth aspect.

In an eighth aspect, an electronic device is provided, and the electronic device includes the board provided in the seventh aspect.

It can be seen that the embodiment of the present disclosure has the advantage of implementing the convolution operation and the activation operation with a single instruction, so it has the advantages of reducing calculation time and saving power consumption.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present disclosure more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a schematic structural diagram of a computing device provided by the present disclosure.

FIG. 2 is a schematic block diagram of an interconnection module provided by an embodiment of the present disclosure.

Fig. 2a is a schematic block diagram of a main operation module in an apparatus for performing forward operation of a convolutional neural network provided by an embodiment of the present disclosure.

Fig. 2b is a schematic block diagram of a slave operation module in the apparatus for performing forward operation of a convolutional neural network provided by an embodiment of the present disclosure.

FIG. 3 is a flowchart of executing a convolution transformation instruction by a convolutional neural network computing device provided in an embodiment of the present disclosure.

FIG. 3a is a schematic diagram of a convolution kernel provided by an embodiment of the present disclosure.

FIG. 3b is a schematic diagram of input data provided by an embodiment of the present disclosure.

FIG. 3c is a schematic diagram of movement of a convolution kernel provided by an embodiment of the present disclosure.

FIG. 3d is a schematic diagram of another movement of a convolution kernel provided by an embodiment of the present disclosure.

FIG. 3e is a schematic diagram of still another convolution kernel movement provided by an embodiment of the present disclosure.

Detailed ways

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, but not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present disclosure.

The terms "first," "second," "third," and "fourth" in the description and claims of the present disclosure and the accompanying drawings are used to distinguish different objects, rather than to describe a specific order. . Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes For other steps or units inherent to these processes, methods, products or devices.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present disclosure. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

The convolution instruction operation method is described below by taking the convolution operation instruction as an example. The convolution instruction can be applied in neural networks. Of course, in practical applications, it can also be applied in other computing scenarios. The present disclosure does not limit the above The specific implementation scenario of the convolution instruction, the convolution operation instruction may also be called a convolutional neural network. For the convolution instruction, the actual formula that needs to be executed can be s=s(∑wx _i +b) where, the convolution kernel w (which may include multiple data) is multiplied by the input data _xi ,

After summing, the initial calculation result h can be obtained by adding the offset b according to the actual calculation, and then the activation operation s(h) can be performed on the preliminary calculation result to obtain the final output result S. According to the formula, the calculation topology can be obtained as multiplier-adder-activation operator.

For the existing convolution instruction, if the activation operation needs to be executed, it needs to be executed through multiple instructions. Taking the above formula as an example, first, it needs to obtain the preliminary calculation result h through the convolution operation instruction, and then activate the convolution through the convolution instruction. The instruction performs an activation operation on this h, that is, at least two convolution instructions are required to obtain the result S of the above formula. This method requires multiple numbers for the number of convolution instructions. In addition, for the chip or computing device, due to the The data needs to be called repeatedly, so it requires more computational overhead and higher power consumption.

The present disclosure provides a computing device. As shown in FIG. 1 , the computing device includes: a storage medium 111, a register unit 112, an interconnection module 113, an arithmetic unit 114, a controller unit 115, and a data access unit 116;

The operation unit 114 may include: a multiplier and an adder, and of course, the operation unit may also include at least one of a comparator, an activation operator, and an OP converter.

The interconnection module 113 is used to control the connection relationship of the calculators in the operation unit 114 so that at least two kinds of calculators form different computing topology structures.

The register unit 112 is used to store operation instructions, input data, addresses of the convolution kernels in the storage medium, and calculation topology structures corresponding to the convolution instructions.

The storage medium 111 can be an off-chip memory, and of course, in practical applications, it can also be an on-chip memory for storing input data and convolution kernels, and the input data and convolution kernels can specifically be vectors, matrices, or multi-dimensional data.

The controller unit 115 is used to extract an operation instruction (specifically, a convolution instruction), an operation domain corresponding to the operation instruction, and a first calculation topology corresponding to the operation instruction from the register unit 112, and decode the operation instruction into a The execution instruction is used to control the operation unit to perform the operation operation, transmit the operation domain to the data access unit 116 , and transmit the calculation topology structure to the interconnection module 113 .

The data access unit 116 is configured to extract the input data and the convolution kernel corresponding to the operation domain from the storage medium 111 , and transmit the input data and the convolution kernel to the operation unit 114 .

The interconnection module 113 is configured to form a first computing topology structure according to the connection relationship of the calculators in the control operation unit 114 .

The operation unit 114 is configured to call the calculator to perform an operation on the data block according to the first computing topology and the execution instruction to obtain an operation result, and transmit the operation result to the data access unit for storage in the storage medium.

The operation instruction can be shown in Figure 1, including: an operation field and an operation code. Taking the convolution operation instruction as an example, the operation field can include: a convolution subfield and an activation subfield, as shown in Table 1, where the register number (Optional, the register can also be a register file) 0, register number (optional, register file) 1, register number (optional, register file) 2, register number (optional, register file) 3, Register number 4, register number 0, register number 1, register number 2, and register number 3 may be convolution subfields. Specifically, register number 4 may be an activation subfield.

Table 1:

When it is the activation function interpolation table address, for the computing device, it can save the setting of the activation calculator, and the setting of the activation function interpolation table address can also save the decoder parsing overhead, reduce the calculation amount, and save the chip power consumption and area. . The specific implementation method is described in detail below. For example, if CONV_ACTIVATE contains the address of the activation function interpolation table, the CONV_ACTIVATE command obtains the result of the convolution operation (ie the intermediate result) after the convolution operation is performed, and then extracts the address corresponding to the activation function interpolation table address. The interpolation table performs the activation operation on the result of the convolution operation to get the result directly. This method only needs to read the CONV_ACTIVATE instruction once, and the execution does not require a separate activation calculator to execute, so it has the advantages of low cost of parsing instructions, reducing the amount of calculation, and saving hardware configuration. If CONV_ACTIVATE contains the activation function opcode, the CONV_ACTIVATE instruction After the convolution operation is performed, the result of the convolution operation is obtained, the operation code of the activation function is parsed to obtain the corresponding activation function, and then the activation function is sent to the activation calculator, and the activation calculator extracts the interpolation table according to the activation function. The result of the product operation performs the activation operation, which requires parsing multiple instructions, and additionally requires a separate activation calculator to perform the activation operation.

The operation instruction may also be shown in Table 2, including: opcode CONV_AC_OP, register number (optional, register file) 0, register number (optional, register file) 1, register number (optional, register file) )2, register number (optional, register file) 3, register number (optional, register file) 4, auxiliary opcode, register number 0, register number 1, register number 2, register number 3 can be convolution Subfield, register number 4 can be the activation subfield, and the OP operation code can be the OP subfield, specifically, as shown in Table 2.

Table 2:

The above operation instruction may include a convolution instruction set, the instruction set includes the convolutional neural network CONV instruction, CONV_ACTIVATE instruction, CONV_OP and CONFIG instruction, IO instruction, NOP instruction, JUMP instruction, and MOVE instruction with different functions.

The auxiliary operation codes shown in Table 1 and Table 2 may specifically include calculation operations and calculator connection relationships. Taking the OP operation as an example, there are many types of OP operations, assuming that 1 means transposition and 0 means conjugation. It is assumed that the auxiliary opcode can be 4bit, and in practical applications, it can also be other bit numbers, such as 6bit, 8bit, etc. Etc., for the auxiliary opcode of CONV_OP, if it is 1111, it can be expressed as a transposition operation. The transposition operation that needs to be performed can include: input data, convolution kernel, and preliminary calculation results. Here, it is assumed that the second bit of 1111 Indicates whether the input data performs the OP operation, the third bit indicates whether the convolution kernel performs the OP operation, and the fourth bit indicates whether the preliminary calculation result performs the OP operation. Suppose 1 can be used to perform the OP operation, and 0 can be used to not perform the OP operation. . Of course, other operations are also possible in practical applications.

In one embodiment, the CONV_ACTIVATE instruction includes:

The convolution activation command, according to the command, the device fetches the input data and the convolution kernel of the set size from the specified address of the memory (preferably, the high-speed temporary storage memory) respectively, performs the convolution operation in the convolution operation unit, and then Perform the activation function operation on the output result; the above setting size can be defined by the manufacturer or the user.

Convolution activation instructions can specifically include:

The convolutional neural network Maxout instruction may specifically include: the device takes out the input data and convolution kernel of the set size from the specified address of the memory (preferably, the high-speed temporary storage memory) respectively, and performs the convolution operation in the convolution operation component. , and then activate the output result as Maxout; the above set size can be defined by the manufacturer or the user. The specific expression form of the Maxout instruction for the convolutional neural network may be, adding the Maxout interpolation table or the Maxout opcode to the register number 4 of the operation domain of the CONV_ACTIVATE instruction.

For Maxout, its mathematical expression can be:

whereZ _ij =x ^T W _ij +b _ij

Among them, h _i represents the output result of Maxout, W _ij represents the convolution kernel, b _ij represents the bias, and X ^T represents the transposition of the input data.

The convolutional neural network PReLU instruction is used to perform PReLU activation on the output result of the computing device according to the instruction. Do the convolution operation, and then activate the output result by PReLU; for the specific representation of the convolutional neural network PReLU instruction, add the PReLU interpolation table or PReLU opcode to register number 4 of the operation domain of the CONV_ACTIVATE instruction.

The convolutional neural network RReLU instruction is used to perform RReLU activation on the output result of the computing device according to the instruction, and the device fetches the input data and convolution kernel of the set size from the specified address of the high-speed temporary storage memory, respectively, in the convolution operation unit Do the convolution operation, and then activate the output result by RReLU; for the specific representation of the convolutional neural network RReLU instruction, add the RReLU interpolation table or RReLU opcode to the register number 4 of the operation domain of the CONV_ACTIVATE instruction.

The convolutional neural network Leaky ReLU command is used to activate LeakyReLU on the output result of the computing device according to the command. Do the convolution operation in , and then activate the output result by Leaky ReLU; for the specific expression of the Leaky ReLU instruction of the convolutional neural network, the interpolation table of RReLU or the Leaky ReLU operation can be added to the register number 4 of the operation field of the CONV_ACTIVATE instruction. code.

For ReLU, its mathematical expression is: f(X)=max(0,x);

Leaky ReLU, RReLU, PReLU, its mathematical expression can be:

f(X)=αx(x<0), f(X)=x(x≥0);

For the above mathematical expressions, different values of α correspond to Leaky ReLU, RReLU or PReLU. When α>0, it is PReLU; when α<0, it is Leaky ReLU, and when α is a random number of Gaussian distribution, it is RReLU.

The CONV_ACTIVATE instruction can also include other operation instructions to perform nonlinear activation or linear activation operations.

In one embodiment, the CONV_OP instruction includes:

The convolution transformation instruction, according to the instruction, the device fetches the input data and convolution kernel of the set size from the specified address of the memory (preferably, the cache memory) respectively, in the OP (conjugate or transpose) operation part Transform the input data and/or the convolution kernel, then perform the convolution operation in the convolution operation unit, and then transform the output result; the above set size and OP type can be defined by the manufacturer or the user.

The convolution transformation instructions specifically include:

The convolutional neural network Reshape command is used to perform a Reshape operation on the output result of the computing device according to the command. The device retrieves the input data and convolution kernel of the set size from the specified address of the memory (preferably, the cache memory). , perform the reshape (dimension reorganization, such as nchw->chwn, etc.) operation in the OP operation unit, then perform the convolution operation in the convolution operation unit, and then perform the reshape operation on the output result; the above setting size can be set by the manufacturer or User-defined.

The so-called dimension reorganization refers to the reorganization of the input data of the convolution operation and the four dimensions of the convolution kernel.

For the M convolution kernels shown in Figure 3a, each convolution kernel is a 5*3*3 three-dimensional data block, then its operation window is also a 5*3*3 three-dimensional data block. For the three-dimensional data block shown in Figure 3a The KH and KW in the M convolution kernels indicate that the dimension corresponding to the KH is the H dimension of the input data, and the corresponding dimension indicated by the KW is the W dimension of the input data. The gray blocks in Figures 3c, 3d, and 3e are the data used for each sliding operation window, and the sliding direction can take H as the sliding direction and then use W as the sliding direction or W as the sliding direction. H is the sliding direction. Specifically, for convolution, the operation at each sliding window is that the data block represented by the gray block in the figure and the M convolution kernel data blocks represented by "Convolution 1-convolution kernel in Figure 3a" are performed separately. Inner product operation, the convolution will output a value for each convolution kernel corresponding to each sliding window position, that is, there are M output values for each sliding window; a square is used in the "Figure 3a-Figure 3e" to represent A numerical value can also be called a weight; the numbers used in the schematic diagram are only for illustration, and in actual situations, the dimension data may be any numerical value (including the case where a certain dimension is 1, in this case, the four-dimensional The data block automatically becomes a three-dimensional data block. For example, when the number of samples calculated at the same time is 1, the input data is a three-dimensional data block; for example, when the number of convolution kernels is 1, the convolution and data are a 3D data blocks). Use the chip device to perform a convolution operation between the input data B and the convolution kernel A;

For a convolutional layer, its weights (all convolution kernels) are shown in "Fig. 3a Convolution 1-convolution kernel", and the number of convolution kernels is M, and each convolution kernel consists of C KH It consists of a matrix of rows, KW, and columns, so the weight of the convolutional layer can be expressed as a four-dimensional data block with four dimensions of M, C, KH, and KW; the input data of the convolutional layer is a four-dimensional data block, which consists of N three-dimensional data blocks. data blocks, each of the three-dimensional data blocks is composed of C feature matrices with H rows and W columns (that is, the four dimensions are respectively N, C, H, and W data blocks);

The convolutional neural network Pad instruction is used to perform a Pad operation on the output result of the computing device according to the instruction, and the device fetches the input data and convolution kernel of the set size from the specified address of the memory (preferably, the cache memory) respectively. , perform the pad (peripheral amplification) operation on the convolution kernel in the OP operation unit, and then perform the convolution operation in the convolution operation unit; the above setting size can be defined by the manufacturer or the user. The specific expression form of the Pad instruction of the convolutional neural network may be that the Pad opcode is added to the auxiliary opcode of the operation domain of the CONV_OP or CONV_AC_OP instruction.

Peripheral amplification refers to adding N circles of data to the periphery of the convolution kernel, where N is a positive integer. N can be 1. At this time, the format of this instruction remains unchanged. The circle means that the original two-dimensional data block of kh*kw size is expanded to (kh+2N)*(kw+2N) by the peripheral complement.

If N is greater than 1, either an operation field (register 5) is added to the instruction format to store the value of n, that is, a register 5 is added to the operation field of CONV_OP, and this register 5 is used to store the value of n. If the instruction format is unchanged, the method of executing the instruction is changed. Before executing the CONV instruction, use the config instruction to call the value of n, and execute the pad operation before executing the CONV instruction.

In addition, the data can be all 0, which is the most basic pad operation.

Optionally, the data can be randomly distributed with 0s and 1s. In this case, the opcode has to be changed to conv-pad-random. The method is one more step: use a random number generator to generate the number that pad needs to be filled, a total of (kh+2N)*(kw+2N)–kh*hw data.

The convolutional neural network Crop command is used to perform a Crop operation on the output result of the computing device according to the command, and the device respectively fetches the input data and convolution kernel of the set size from the specified address of the memory (preferably, the cache memory). , Crop (size crop) the input in the OP operation unit, and then perform the convolution operation in the convolution operation unit. The above set size can be defined by the manufacturer or the user.

The definition of size cropping is to cut out a two-dimensional data block of size H1*W1 from the original two-dimensional data block of H*W size, where H1 and W1 are less than or equal to H and W.

The convolutional neural network Dilate instruction is used to perform a Dilate operation on the output result of the computing device according to the instruction, and the device fetches the input data and the convolution kernel of the set size from the specified address of the memory (preferably, the cache memory) respectively. , perform the dilate (internally insert 0) operation on the convolution kernel in the OP operation unit, and then perform the convolution operation in the convolution operation unit; the above setting size can be defined by the manufacturer or the user.

The definition of dilate (internal insertion of 0) is: for the convolution kernel of kh*kw size, insert 0 or random numbers uniformly or randomly in its interior (the pad mentioned above is on the periphery), which plays a role in the convolution kernel " Dilution" effect, which can enhance the feature extraction effect of the convolution kernel.

The CONV_OP instruction can also include other transformation instructions, such as BLAS transformation on the input and weights.

The above instruction set includes the convolutional neural network CONV_AC_OP instruction with different functions, as well as the CONFIG instruction, the IO instruction, the NOP instruction, the JUMP instruction and the MOVE instruction.

In one embodiment, CONV_AC_OP can implement any combination of CONV, ACTICATE and OP operations through the setting of auxiliary opcodes.

Figure 2 schematically shows one embodiment of the interconnection module 113: an H-tree module. The interconnection module 113 constitutes a data path between the master operation module 5 and a plurality of slave operation modules 6, and is a binary tree path composed of multiple nodes. The data returned by the two downstream nodes is merged and returned to the upstream node. For example, in the calculation stage of the convolutional neural network, the neuron data in the main operation module 5 is sent to each slave operation module 6 through the interconnection module 4; when the calculation process of the slave operation module 6 is completed, when the calculation process of the slave operation module After the process is completed, the value of each neuron output from the operation module will be assembled into a complete vector composed of neurons in the interconnection module. For example, assuming that there are N slave operation modules in the device, the input data Xi is sent to N slave operation modules, and each slave operation module performs a convolution operation on the input data Xi and the corresponding convolution kernel of the slave operation module, A scalar data is obtained, and the scalar data of each slave operation module is combined by the interconnection module 4 into an intermediate vector containing N elements. Assuming that the convolution window is traversed to obtain A*B pieces of input data Xi (A pieces in the X direction, B pieces in the Y direction, X, Y are the coordinate axes of the three-dimensional orthogonal coordinate system) input data Xi, then perform the above execution on the A*B pieces of Xi. Convolution operation, all the obtained vectors are combined in the main operation module to obtain the three-dimensional intermediate result of A*B*N.

Fig. 2a shows an example block diagram of the structure of the main operation module 5 in the apparatus for performing forward operation of a convolutional neural network according to an embodiment of the present disclosure. As shown in FIG. 2 b , the main operation module 5 includes a first operation unit 51 , a first data dependency relationship determination unit 52 and a first storage unit 53 .

The first operation unit 51 includes a vector addition unit 511 and an activation unit 512 . The first operation unit 51 receives the control signal from the controller unit and completes various operation functions of the main operation module 5. The vector addition unit 511 is used to realize the bias addition operation in the forward calculation of the convolutional neural network. The offset result is obtained by bitwise addition of the set data and the intermediate result, and the activation operation unit 512 performs an activation function operation on the offset result. The offset data may be read in from an external address space or stored locally.

The first data dependency determination unit 52 is a port for the first operation unit 51 to read and write the first storage unit 53 , and ensures the read-write consistency of the data in the first storage unit 53 . At the same time, the first data dependency determination unit 52 is also responsible for sending the data read from the first storage unit 53 to the slave operation module through the interconnection module 4, and the output data from the operation module 6 is directly sent to the slave operation module through the interconnection module 4. The first arithmetic unit 51 . The instructions output by the controller unit 2 are sent to the calculation unit 51 and the first data dependency determination unit 52 to control their behavior.

The storage unit 53 is used for buffering the input data and output data used by the main operation module 5 in the calculation process.

FIG. 2b shows an example block diagram of the structure of the slave operation module 6 in the apparatus for performing forward operation of a convolutional neural network according to an embodiment of the present disclosure. As shown in FIG. 2A , each slave operation module 6 includes a second operation unit 61 , a data dependency relationship determination unit 62 , a second storage unit 63 and a third storage unit 64 .

The second operation unit 61 receives the control signal sent by the controller unit 2 and performs a convolution operation. The second operation unit includes an OP transformation unit 808, a vector multiplication unit 611 and an accumulation unit 612, which are respectively responsible for vector multiplication operation, accumulation operation and OP transformation operation in the convolution operation.

The second data dependency determination unit 62 is responsible for read and write operations on the second storage unit 63 in the calculation process. Before the second data dependency determination unit 62 performs the read and write operations, it will firstly ensure that there is no read-write consistency conflict in the data used between the instructions. For example, all control signals sent to the data dependency unit 62 will be stored in the command queue inside the data dependency unit 62. In this queue, if the range of the read data of the read command is the same as that of the write command at the front of the queue If the range of write data conflicts, the instruction must wait until the dependent write instruction is executed before it can be executed.

The second storage unit 63 buffers the input data and output scalar data of the slave operation module 6 .

The third storage unit 64 buffers the convolution kernel data required by the slave operation module 6 in the calculation process.

FIG. 3 is a flowchart of a convolutional neural network computing device according to an embodiment of the present disclosure executing a convolutional transformation instruction. As shown in FIG. 3 , the process of executing a convolutional neural network instruction includes, where the convolutional neural network instruction is: CONV_AC_OP For example, in practical applications, it can also be other extended instructions, such as CONV_ACTIVATE or CONV_OP instruction: if the extended instruction is a CONV_OP instruction, only the OP operation needs to be performed, and there is no need to perform the activation operation on the offset data in S9. , that is, if the extended instruction is the CONV_OP instruction, the offset data is the final output result. If the extended instruction is the CONV_ACTIVATE instruction, the operation device does not need an OP module, and also does not need to perform OP conversion in step S7.

In step S1, an IO instruction is pre-stored at the first address of the register unit 112.

In step S2, the operation starts, the controller unit 115 reads the IO instruction from the first address of the register unit 112, and according to the decoded control signal, the data access unit 116 reads all corresponding convolutional neural network operations from the storage medium 111 instruction and cache it in register unit 112.

In step S3, the controller unit 115 reads the next IO instruction from the register unit 11, and according to the decoded control signal, the data access unit 116 reads all the data (for example, including the input data) required by the main operation module 5 from the storage medium 111 , an interpolation table for fast activation function operation, a constant table for configuring the parameters of the operation device, offset data, etc.) to the first storage unit 53 of the main operation module 5 .

In step S4 , the controller unit 115 reads the next IO instruction from the register unit 11 , and according to the decoded control signal, the data access unit 116 reads the convolution kernel data required from the operation module 6 from the storage medium 111 .

In step S5, the controller unit 115 reads the next CONFIG instruction from the register unit 11, and according to the decoded control signal, the device configures various constants required for the calculation of the neural network of this layer. For example, the first operation unit 51 and the second operation unit 61 configure the value of the internal register of the unit according to the parameters in the control signal, and the parameters include, for example, the data required by the activation function; and the constants required for the OP operation, such as pad N, H1 and W1 of crop, dimension order of reshape, etc.

In step S6, the controller unit 115 then reads the next CONV_AC_OP instruction from the register unit 11, and according to the decoded control signal, the master operation module 5 first sends the input data in the convolution window to each slave operation module through the interconnection module 113 6. Save to the second storage unit 63 of the slave operation module 6, and then move the convolution window according to the instruction.

In step S7, according to the control signal decoded by the CONV_AC_OP instruction, the convolution kernel is read from the third storage unit 64 from the operation unit 61 of the operation module 6, and the input data is read from the second storage unit 63, and the OP module is used for the input data. And the convolution kernel is changed by OP, and then the operation unit 61 of the operation module 6 performs the convolution operation of the input data (OP transformation) and the convolution kernel (OP transformation), and returns the intermediate result through the interconnection module 113 .

In step S8, in the interconnection module 113, each intermediate result returned from the operation module 6 is spliced into a complete intermediate vector step by step.

In step S9, the main operation module 5 obtains the intermediate vector returned by the interconnection module 4, the convolution window traverses all input data, and the main operation module splices all the returned vectors into an intermediate result; (optional) Control according to the CONV_AC_OP instruction signal, read the offset data from the first storage unit 53, add the intermediate result to the offset result through the vector addition unit 511, and the main operation module 5 reads the interpolation table corresponding to the activation function interpolation table address in the CONV_AC_OP register number 4, Activate the offset result and the interpolation table to obtain the final output data, and write the final output data back to the first storage unit 53 .

In step S10, the controller unit 115 then reads the next IO instruction from the instruction storage unit, and according to the decoded control signal, the data access unit 116 stores the output data in the first storage unit 53 to the specified address in the external address space, and calculates Finish.

Embodiments of the present disclosure further provide a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, the computer program causing the computer to execute any one of the convolution expansion instructions described in the above method embodiments. Implement some or all of the steps of the method.

Embodiments of the present disclosure also provide a computer program product, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, the computer program being operable to cause a computer to execute the method described in the foregoing method embodiments Some or all of the steps in the implementation of any convolution expansion instruction.

Another embodiment of the present disclosure further discloses a chip, which includes the neural network computing device of the above embodiment (as shown in FIG. 1 ).

Another embodiment of the present disclosure further discloses a chip package structure including the above-mentioned chip.

Another embodiment of the present disclosure further discloses a board including the above-mentioned chip packaging structure.

Another embodiment of the present disclosure further discloses an electronic device including the above board.

Electronic devices include data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, cloud servers, cameras, video cameras, projectors, watches, headphones, mobile Storage, Wearables Vehicles, Home Appliances, and/or Medical Devices.

The vehicles include airplanes, ships and/or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lamps, gas stoves, and range hoods; the medical equipment includes nuclear magnetic resonance instruments, B-ultrasound and/or electrocardiograph.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the described action sequence. As in accordance with the present disclosure, certain steps may be performed in other orders or concurrently. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The embodiments of the present disclosure have been introduced in detail above, and the principles and implementations of the present disclosure are described in this paper by using specific examples. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present disclosure; at the same time, for Persons of ordinary skill in the art, according to the idea of the present disclosure, will have changes in the specific implementation manner and application scope. In conclusion, the contents of this description should not be construed as a limitation of the present disclosure.

Claims (13)

1. a kind of execution method of convolution extended instruction, which is characterized in that described method includes following steps:

Computing device reads the convolution extended instruction from memory and obtains the input data of the convolution extended instruction, convolution kernel And activation operation；

The convolution extended instruction includes: operation code and operation domain, and the operation code includes: the mark of the convolution extended instruction Know；The operation domain includes: convolution subdomain and activation subdomain, and the convolution subdomain includes: address and the volume for storing input data The address of product core, the activation subdomain include: the identification code of the activation operation or the interpolation table address of the activation operation；

Computing device executes convolution algorithm to the input data and convolution kernel and obtains intermediate result, passes through the activation subdomain pair The intermediate result executes activation operation and obtains the final result of described instruction.

2. the method according to claim 1, wherein

The activation operation includes: convolutional neural networks Maxout operation, convolutional neural networks PReLU operation, convolutional Neural net Network RReLU operation, convolutional neural networks Leaky ReLU operation, nonlinear activation operation or linear activation operation operation.

3. the method according to claim 1, wherein if the activation subdomain includes: the interpolation table of activation operation Address, described executed by the activation subdomain to the intermediate result are activated operation to obtain the final result of described instruction, are wrapped It includes:

Computing device extracts the corresponding interpolation table of interpolation table address of the activation operation, by the intermediate result and the interpolation Table executes activation operation and obtains the final result of described instruction.

4. the method according to claim 1, wherein as it is described activation subdomain include: activation operation identification code, Described executed by the activation subdomain to the intermediate result activates operation to obtain the final result of described instruction, comprising:

Computing device identifies that the identification code of the activation operation determines the activation operation, reads the interpolation of the activation operation The interpolation table and the intermediate result are executed activation operation and obtain the final result of described instruction by table.

5. the method according to claim 1, wherein the computing device holds the input data and convolution kernel Row convolution algorithm obtains intermediate result, comprising:

The input data is split into multiple portions and obtains multiple input subdatas by the main computing module of computing device, will be multiple Input subdata be distributed to it is multiple from computing module, convolution kernel is sent to it is multiple from computing module, it is the multiple from operation mould Block executes input subdata parallel and the multiplying of convolution kernel obtains multiple sons as a result, the main computing module of computing device is by institute Multiple sub- results are stated to splice to obtain the intermediate result.

6. a kind of computing device, which is characterized in that the computing device includes: memory, arithmetic element, interconnection module, operation Unit, controller unit and data access unit；

Wherein, the arithmetic element, comprising: adder calculator, multiplicative operator；

Controller unit, for reading the input number that the convolution extended instruction obtains the convolution extended instruction from memory According to, convolution kernel and activation operation；

The convolution extended instruction includes: operation code and operation domain, and the operation code includes: the mark of the convolution extended instruction Know；The operation domain includes: convolution subdomain and activation subdomain, and the convolution subdomain includes: address and the volume for storing input data The address of product core, the activation subdomain include: the identification code of the activation operation or the interpolation table address of the activation operation；

Data access unit, for obtaining the corresponding input data in address and volume of the address and convolution kernel of the input data Product core；

The arithmetic element obtains intermediate result for executing convolution algorithm to the input data and convolution kernel, by described Activation subdomain executes activation operation to the intermediate result and obtains the final result of described instruction.

7. computing device according to claim 6, which is characterized in that

The activation operation includes: convolutional neural networks Maxout operation, convolutional neural networks PReLU operation, convolutional Neural net Network RReLU operation, convolutional neural networks Leaky ReLU operation, nonlinear activation operation or linear activation operation operation.

8. computing device according to claim 6, which is characterized in that if the activation subdomain includes: inserting for activation operation It is worth table address；

The data access unit, for extracting the corresponding interpolation table of interpolation table address of the activation operation；

The arithmetic element obtains the final of described instruction for the intermediate result to be executed activation operation with the interpolation table As a result.

9. computing device according to claim 6, which is characterized in that if the activation subdomain includes: the mark of activation operation Know code；The arithmetic element further include: activation arithmetic unit；

The controller unit, the identification code of the activation operation determines the activation operation for identification；

The activation arithmetic unit executes the interpolation table and the intermediate result for taking the interpolation table of the activation operation Activation operation obtains the final result of described instruction.

10. computing device according to claim 8, which is characterized in that the arithmetic element further include: main computing module and It is multiple from computing module, the main computing module includes: adder calculator and multiplicative operator, described to include: from computing module Adder calculator and multiplicative operator；

The main computing module obtains multiple input subdatas for the input data to be split into multiple portions, will be multiple Input subdata be distributed to it is multiple from computing module, convolution kernel is sent to it is multiple from computing module, it is the multiple from operation mould Block, the multiplying for executing input subdata and convolution kernel parallel obtain multiple sons as a result, the main computing module, is used for Splice the multiple sub- result to obtain the intermediate result.

11. a kind of computer readable storage medium, which is characterized in that it stores the computer program for being used for electronic data interchange, Wherein, the computer program makes computer execute the method according to claim 1 to 5.

12. a kind of computer program product, which is characterized in that the computer program product includes storing computer program Non-transient computer readable storage medium, the computer program are operable to that computer is made to execute such as claim 1-5 Method described in one.

13. a kind of electronic device, which is characterized in that the electronic device includes processor, and the processor includes that right such as is wanted Seek computing device described in 6-10 any one.