An Intermediate-Centric Dataflow for Transposed Convolution Acceleration on FPGA

The transposed convolution algorithm is depicted in Figure 1. The nested loops and MAC operations demonstrate its computation intensiveness. The process of transposed convolution is illustrated in Figure 1: (i) We multiply each input feature element by the weight kernel, generating an intermediate patch. The size and the number of intermediate patches equal the size of the weight kernel and the number of input feature elements, i.e., \(K \cdot K\) and \(I_{W} \cdot I_{H} \cdot I_{C}\), respectively. Then, we should accumulate the intermediate patches produced from multiple input pixels with identical \((I_{h}, I_{w})\) index to flatten the input channel dimension.(ii) It can be seen that these flattened intermediate patches overlap with adjacent ones in both vertical and horizontal directions since \({S}\) is typically smaller than \({K}\). It is necessary to add up the overlapped parts in these two directions as depicted in colorful areas in Figure 1, leading to an output feature map of size \(O_H \cdot O_W\). This is the backward-stencil computation of transposed convolution, and we will present the proposed solution in the Section 3. Generally, the overlapped red and blue pixels are produced separately by horizontally and vertically adjacent intermediate patches, which are generated separately by horizontally and vertically adjacent input feature pixels. Besides, there exist the overlapped purple output pixels that are generated together by adjoining intermediate patches within both directions.

Then, we mainly explain the non-uniform dependence distance incurred by backward-stencil computation. Typically, the dependence distance denotes the distance between dependent iterations, e.g., i -> i + 1 and i -> 2i in the polyhedral model. Initially, we attempt to map the transposed convolution layer in the systolic array as the convolution [17], leveraging the state-of-the-art compiler AutoSA [16]. However, the mapping breaks down owing to the non-uniform dependence.

When mapping the programs containing nested loops into the standard systolic array leveraging polyhedral model [3], we generally select two loops as the space loops and one loop as the time loop. We should keep the dependence (flow, anti, output, and read) distances constant. And the dependence distances on space loops should be no larger than 1 so that data communication just occurs between adjacent Processing Elements (PEs). In the transposed convolutional layer of Figure 1, we can assign one of \(k_{h}\), \(k_{w}\), and \(o_{c}\) as the space loop for accessing weights. And the read-after-read (RAR) dependence distances of these assignments can be 1. Similarly, we can also assign one of \(i_{h}\) and \(i_{w}\) as space loop for accessing input feature maps and the corresponding RAR dependence distances are 1. Then, we set \(i_{c}\) as the time loop so that all the PEs perform MAC operations in the input channel dimension. Consequently, we can map the transposed convolution into the systolic array as plotted in Figure 2(a), where the weight elements and input feature maps can be reused along input height dimension and kernel height dimension, respectively. And the array size can be \(I_{H} \cdot K_{H}\) supposing that there are abundant hardware resources.

As adopting the mapping strategy as shown in Figure 2(a) in which \(S\) can be set as 2, we separately feed one column of input feature map and weight into the array. The input feature element \((i_{h}, i_{w})\) will perform MAC operations with weight element \((k_{h}, k_{w})\) at the PE\((i_{h}, k_{h})\), in which the intermediate MAC result is just a part of the final output result \(o(S \cdot i_{h} + k_{h}, S \cdot i_{w} + k_{w}, o_{c})\). However, as for the next input feature element \((i_{h^{\prime }}, i_{w^{\prime }})\) in which \(i_{h^{\prime }} = i_{h}+1\) and \(i_{w^{\prime }}= i_{w}\) as shown in Equation (3), it will perform MAC operation with weight element \((k_{h^{\prime }}, k_{w^{\prime }})\), in which \(k_{h^{\prime }}=k_{h}-S\) and \(k_{w^{\prime }}=k_{w}\) as shown in Equation (4). And the produced intermediate MAC result should add up with the intermediate one produced in the PE\((i_{h}, k_{h})\) for final result, as shown in Figure 2(b). That is, while computing the output result, the distance between two dependent \(i_{h}\) iterations is uniform, i.e., \(i_{h}\) -> \(i_{h}+1\). However, the distance between dependent \(k_{h}\) iterations change from \(k_{h}\) -> \(k_{h}+1\) to \(k_{h}\) -> \(k_{h}-S\), resulting in the non-uniform RAW dependence. And it breaks the second constraint [3, 16] when leveraging the polyhedral model to map the program into systolic array.

As can be seen, the backward-stencil computation introduces the irregular computation pattern and non-uniform dependence, severely limiting the parallel computation of transposed convolution. To perform the backward-stencil computation or speed up the transposed convolution, the researchers have proposed mainly two kinds of methods. The previous works [5, 12] propose to transform the transposed convolution to multiple convolution operations, which can avoid the backward-stencil computation and adopt the convolution acceleration scheme. However, this method performs the complex transformation in the software and introduces more computation and transformation overhead. On the other hand, the works [15, 19, 20] store temporarily the intermediate patches by leveraging the enormous on-chip memory to perform the backward-stencil computation.

To perform the backward-stencil computation of transposed convolution, we propose the intermediate-centric dataflow scheme that decouples the generation of intermediate patches from the further process on the intermediate ones, achieving the overall transposed convolution. The proposed dataflow typically includes two phases: Flattening Phase and Overlapping Phase, as shown in Figure 3. The Flattening Phase generates the intermediate patch elements by multiplying input feature elements by weights. Then, we transfer and accumulate the overlapped intermediate patch elements in the Overlapping Phase. Firstly, we feed the weight kernel and \((N-1)\)-th row of input feature map with \(I_{c}\) input channels into the accelerator array. In the Flattening Phase, we multiply and accumulate the input feature map with weights to generate the intermediate patch. And these intermediate patch elements plotted in dark blue in the Figure 3 are flattened to distribute in the array PEs. At the moment, these intermediate patch elements can be ready for the next Overlapping Phase, and we can concurrently feed the \(N\)th row of input feature map into the array in the Flattening Phase. Regarding the intermediate patches in the Overlapping Phase, we move and add up the intermediate patch elements that are stored in the PEs linked by green oblique line in the first stage. Then, we continue to move the vertically overlapped intermediate patch elements between PEs linked by \(purple\) vertical line in the second stage. Generally, we can pipeline the Flattening Phase and Overlapping Phase for distinct rows of the input feature map.

We obtain the intermediate results by performing MAC operations between input feature maps and weights from different input channels in the Flattening Phase. As we map kernel height and kernel width dimensions to array rows, input width dimension to array columns, and output channel dimension to PEs, we multiply each input feature element by multiple weight kernels. The data reuse of input feature elements and weights can be achieved in vertical and horizontal directions, respectively.

As shown in Equation (5), the weight streams containing \(w(m, n, 0), \ldots , w(m, n, i_{c})\) flow horizontally into the \(y\)th array row, where \(y = (m \cdot K + n)\). Each stream element \(w(m, n, i_{c})\) is actually one weight vector, which packs several \(w(m, n, i_{c}, o_{c})\) located in \(o_{c}\)th channels. Correspondingly, input pixel streams including \(a(i_{h}, i_{w}, 0), \ldots , a(i_{h}, i_{w}, i_{c})\), which are within \(i_{h}\)th row of input feature map, flow into \(i_{w}\)th array column. In this way, PEs can simultaneously receive the weight vectors and associated input feature elements and perform MAC operations. We then formulate the computation process of Flattening Phase as Equation (5):in which the MAC result is stored in \({R0}_{x,y}\) of \(\mathit {PE}_{x,y}\). Each time PEs receive the weight vector and input pixel, they perform parallel MAC operations and generate the calculation results.

When input feature elements and weights flow out of the array, the Flattening Phase ends. And we obtain the calculated results stored in \({R0}_{i,j}\) of \({PE}_{i,j}\). Then, the calculated results shift from \(R0\) to \(R3\) for the next operation. As this phase corresponds to step \(i\) mentioned in Section 2, these results are actually intermediate results that form the intermediate patches. In the accelerator architecture, each intermediate patch distributes within one array column during this phase.

Generally, the process of Overlapping Phase contains two stages, i.e., the accumulation stage and transfer stage, in which we separately accumulate the overlapped intermediate results and move the intermediate results upward.

We mainly explain the way to schedule the two phases to perform the transposed convolution in the accelerator architecture. For each row of input feature maps, we multiply the input pixels by weight kernels and generate the intermediate patches in Flattening Phase. Then, the intermediate results move along the output-reuse path to accumulate with others in accumulation stage and transfer stage of the Overlapping Phase. With the output-reuse path and register \(R0\) of PEs, we can perform the Flattening Phase concurrently with another phase. In other words, we perform accumulation stage and transfer stage of the Overlapping Phase of \((p{-}1)\)-th row of input feature maps concurrently while undertaking Flattening Phase of \(p\)th row, keeping the PEs busy and improving the resource utilization. In current hardware design of accelerator array, we separately implement the global FSM and inner FSMs inside PEs to control the overall circuit. The gobal FSM generally contains several states including S_G_IDLE, S_G_IMAGE_START, S_G_NEXT_ROW, S_G_PAUSE, S_G_IMAGE_END, and S_G_END. In this way, we can control feeding the input features and their rows into the accelerator array. Besides, inside the PEs, we also implement the inner FSMs, including mac-operation FSM and accumulation-FSM, to separately control the MAC operations on the input feature maps/weights and the overlap accumulation operations. As we initially devise the specific accelerator array towards each transposed convolutional layers, we feed the bitstream and implement the specific hardware design in the FPGA board leveraging the EDA platform, instead of designing the general instructions. To design the uniform hardware circuit, we will further design the effective instructions in the future works. Typically, in the accelerator arrays towards specific transposed convolutional layers, the FSMs can be slightly different as it takes distinct cycles to finish the MAC operations between input feature elements and weights. Once the multipliers and accumulators generate the intermediate results, which are stored in the R0 registers, they just continue to perform MAC operations on the new input data. And the newly generated intermediate results sequentially add up with R1 and R2 under the control of inner FSM, obtaining the results that can be stored in the R3.

The latency and throughput of each phase and stage are summarized in Table 2. For one row of input feature maps, the latency of the Flattening Phase can be calculated as \(K^{2} + I_{W} + I_{C}\). As we leverage the register \(R3\) along the output-reuse path as router, latency of accumulation stage and transfer stage can be \(2 \cdot (\lceil K/S \rceil - 1) \cdot (S+1)\) and \(2 \cdot K \cdot S\), respectively. Generally, the latency of Flattening Phase is larger than the latency sum of accumulation stage and transfer stage. It means that there is adequate time to transfer intermediate results along the output-reuse path to calculate final overlapped results.

On the other hand, we adopt Initiation Interval (\(II\)), to denote the throughput of these phases and stages. Typically, one row of input feature map that consists of \(I_{C}\) input channels flows into the array each time. It takes \(I_{C}\) cycles to complete the Flattening Phase, after which we can feed the next row. Correspondingly, the \({II}_{f}\) of Flattening Phase is \(I_{C}\). In the accumulation stage, it typically takes \({Latency}_{o}\) cycles to finish shift operations and generate the horizontal overlaps, after which we can execute the Overlapping Phase in the next round. Then, the \({II}_{o}\) can equal \({Latency}_{o}\), e.g., \(2 \cdot (\lceil K/S \rceil - 1) \cdot (S+1)\). Regarding transfer stage, \({II}_{s}\) equals 1 as horizontal overlapped results can move upward immediately after the previous phase ends.

Observing the Equations (9) and (11) of accumulation stage and transfer stage, we find that intermediate elements \({I}_{x}\) can participate in several accumulation operations for overlapped results. From another perspective, \({I}_{x}\) can be reused in several accumulation processes in both phases. To support the intermediate element reuse, we introduce the output-reuse path in the array. And the topological structure of output-reuse path can be determined by Equations (9) and (11). We perform the backward-stencil computation by transferring and accumulating associated intermediate results along the output-reuse path in a regular way while performing MAC operations on input feature maps and weights.

Although the accelerator implementation of intermediate-centric dataflow has been presented, it still takes enormous labours and hours to design the accelerator for target networks under given resources. To shorten the tedious and long-term design process, we propose the automatic framework, as plotted in Figure 9, for automatic implementation of the accelerator array. Initially, the framework takes in the input parameters, including layer parameters and FPGA device type, by which we obtain the resource budget of FPGA devices in the Parameter Analysis module. Then, we can explore the design space in the Design Space Exploration module to obtain the optimal design parameters leveraging the proposed analytical model, by which we estimate the design parameters and expedite the design space exploration. Generally, the parameters \(N_{i}\) and \(N_{o}\) denote the numbers of input feature map and output result. And \(O_{TC}\) denotes the adopted parallel factor within PEs. These parameters together affect the resource cost of the array. Additionally, we assign the parameter \(F_{freq}\) as certain values to help estimate the performance. In the Hardware Generator module, we realize the agile design of the accelerator array based on the design parameter and hardware templates. We devise the parameterized templates of PE in Chisel [1] for the design automation of arrays [7], since the hardware design of PE is fixed and FSM/output-reuse path is regularly determined by the layer parameters towards transposed convolutional layer \((K, S, I_{H}, I_{W}, I_{C}, O_{C})\). Besides, the associated buffers and global controller can be attained and the generated accelerator array can be implemented on the target device with the Xilinx Vitis platform. Then, an end-to-end framework is devised for the automatic generation of accelerator architecture for transposed convolution. Leveraging the automatic framework, we can obtain the optimal accelerator with a given hardware resource.

The problem of accelerator design can be formulated as Equation (12). The \(T_{comp}\) denotes the latency of calculating one input feature map in the array, and \(T_{load}\) denotes the latency of loading one input feature map from off-chip storage. The objective is to find the maximal one between \(T_{comp}\) and \(T_{load}\), then minimize it without exceeding the limitations of on-chip resources. To solve the problem of Equation (12), we further propose the related resource model and performance model as presented below.

We evaluate our intermediate-centric dataflow and corresponding accelerator design in the Alveo U200 board which contains 2,586 K logic cells, 6,840 DSPs, 75.9 Mb blockRAM (BRAM), and 270 Mb UltraRAM (URAM) on the chip. The implementation of hardware accelerator is synthesized by Xilinx Vitis 2020.1 platform. And we select several transposed convolutional layers from widely applied networks, e.g., FSRCNN, DCGAN, FCN, as the benchmark to verify our intermediate-centric dataflow and the accelerator design, as shown in Tables 3 and 4. The FSRCNN used in our experiment contains five layers, in which the last one is transposed convolutional layer that can expand the low-resolution image of the CIFAR-10 and Set5 [2] datasets according to different stride sizes such as 2, 3, and 4. The DCGAN consists of four convolutional layers and four transposed convolutional layers, where all the kernel size is 5. As the work [19], we perform the DCGAN on the LSUN dataset for scene modeling. The FCN has three transposed convolutional layers, where the kernel sizes are separately 4, 4, and 16. And we perform the FCN on the KITTI [19]. We quantize the input feature elements into 16-bit and weight elements into 8-bit, respectively. As the accelerator array is proposed for transposed convolution acceleration, we mainly focus on the transposed convolutional layers within these networks.

In recent years, DCGAN has been proposed to make great progress in the field of generator adversary networks, owing to its unique network structures. Generally, DCGAN consists of four convolutional layers and four transposed convolutional layers without any Fully Connected (FC) layers, and their parameters, as described in Section 2, are listed in Table 3.

Initially, we design the specific accelerator array for each layer within DCGAN. As for the first transposed convolutional layer, whose size equals \(({5, 2, 4, 4, 1024, 512})\), the array rows and columns can be assigned as \({K}^2\) and \({I}_{W}\), i.e., 25 and 4. Then, we can assign the \({O}_{TC}\) as 10 via the analytical model under the DSP resource limitation, considering the routing feasibility. Consequently, we can devise the corresponding accelerator array \((25, 4, 10)\) for the first transposed convolutional layer of DCGAN. To determine the size of Input Buffer and Output Buffer, we then figure out the parameters \(N_i\) and \(N_o\) by solving the problem formulated in Section 4. Generally, the sizes of one input feature map and output result are 256 Kb and 512 Kb, we can set the \(N_i\) and \(N_o\) as 4 from the obtained results of the formulated problem under the memory constraints. Hence, the size of Input Buffer and Output Buffer can be assigned separately as 1 Mb and 2 Mb, and the Weight Buffer size equals 4 Mb with \(O_{TC}\) equaling 10. The performance of the devised accelerator array reaches 1101.34GOPS when we implement it in the U200 platform with 200 MHz frequency, as shown in Table 5.

Then, we present the accelerator array towards the second transposed convolutional layer of DCGAN, whose layer size is \(({5, 2, 8, 8, 512, 256})\). Primarily, the sizes of array row and column can be set as 25 and 8, respectively. Identically, we can also assign the parallel factor \(O_{TC}\) as 10 for routing. In this way, we can obtain the corresponding accelerator array towards the second transposed convolutional layer. As the sizes of the input feature map and output result separately are 512 Kb and 1 Mb, we also set the \(N_i\) and \(N_o\) by the design space exploration. And we implement the Input Buffer, Output Buffer, and Weight Buffer, whose sizes separately equal 1 Mb, 2 Mb and, 2 Mb, in the U200 platform. The performance of devised accelerator array can be 2012.71GOPS.

As for the third layer \(({5, 2, 16, 16, 256, 128})\) within DCGAN, we can generate the corresponding accelerator array leveraging the analytical model. As the kernel size \(K\) equals 5 and the input width \(I_{W}\) equals 16, its output channel \(O_{C}\) reaches 128. It is necessary to adopt tiling in the output channel, i.e., \(O_{TC}\). Due to the resource constraints and fan-out limits, we assign the \(O_{TC}\) as 10 with the help of the analytical model. Then, we still load the input feature maps and weights from off-chip memory in a double-buffer manner to hide the long-latency memory access. Since the \(T_{comp}\) can be larger than \(T_{load}\) as \(O_{TC}\) equals 10, we can assign the \(N_{i}\) and \(N_{o}\) as 2 to satisfy the resource limitation via the analytical model. We can iteratively load the rows of input feature maps to multiply weights concurrently in the output channels. And we implement the Input Buffer, Output Buffer and Weight Buffer, whose sizes separately equal 1 Mb, 2 Mb, and 1 Mb, around the accelerator array. By consuming 4000 DSPs running at 200 MHz, we can achieve 3920.24 GOPS by implementing the accelerator array on the U200 platform, as listed in Table 5.

Towards the fourth layer \(({5, 2, 32, 32, 128, 3})\), we specifically propose the accelerator array that contains 25 rows and 32 columns. Since the number of output channels equals 3, we can assign the parallel factor of PEs as the identical value, without partitioning in the \(O_{C}\) dimension. Generally, the sizes of input feature maps and output results of this layer are 2 Mb and 192 Kb, separately. By solving the formulated problem, we set the \(N_{i}\) and \(N_{o}\) as 2, and implement the 4 Mb Input Buffer and 0.4 Mb Output Buffer, respectively. Besides, we also deploy the Weight Buffer whose size equals 154 Kb. By deploying the designed accelerator array in the target platform, we obtain the acceleration performance 2520.66GOPS.

As can be seen, we firstly select the transposed convolutional layers within DCGAN to demonstrate the effectiveness of accelerator architecture. Typically, since the kernel size and stride size of the four layers are identical, we can only focus on the output channel dimensions in which we apply the tilting method. As we can see, by resolving the min–max problem proposed in the design space problem, we can devise the specific accelerator array with the appropriate tiling factor and buffer size. Benefiting from the systolic way, the devised array can run at a much higher frequency. However, we assign the frequency identical to that adopted in the next section for a fair comparison. Moreover, we can also implement several accelerator arrays concurrently in the selected platform under resource constraints, and map different tiles split in output channel dimension to these arrays. In this way, we can fully utilize the available resource to reap higher performance.

Then, to demonstrate the reuse of the proposed accelerator array, we devise one uniform array in which we can map these transposed convolutional layers. Generally, the size of the uniform array can be set as \((25, 8, 10)\), as the transposed convolutional layers have the same kernel size. This means that we apply tiling in the output channel dimensions of these layers, and the \(O_{TC}\) equals 10. Additionally, we also need apply tiling in the input width dimensions of the \(3rd\) and \(4th\) transposed convolutional layers, i.e., the tiling size of the input width dimension \(I_{TW}\) equals 8. Then, the intermediate results obtained from the tiles split in the input width dimensions should be accumulated to form the final results. By applying tiling in the output channel and input width dimensions, we can execute these transposed convolutional layers in the uniform accelerator array.

We initially re-implement the the basic accelerator design for transposed convolution [18], which are similar to the method in [15, 20], in the Xilinx Alveo U200 platform. Typically, we map the output channel dimension and input width dimension to the array, and leverage the on-chip buffers beside PEs to perform the overlap-and-accumulate operations on the intermediate patches. As the array size equals \(16 \times 16\), it takes up 256 DSPs and achieves 0.32 GOPS/DSP, which can be relatively lower than other works. This is mainly because the generation of the intermediate patches and further backward-stencil operation are executing sequentially. After we finish performing the backward-stencil operations on the intermediate patches that are currently stored in the on-chip buffers, we can only continue to feed the next row of input feature maps to generate the new patches that can be written in the on-chip buffers.

In this section, we investigate the effects of kernel size and stride size on acceleration performance. Generally, we select the third transposed convolutional layer of FCN and the last layer of FSRCNN, both of which have different kernel sizes. Specially, we can assign the stride size of the last layer of FSRCNN as distinct numbers, to figure out the effect of stride size on the accelerator array. And we select the 3rd transposed convolutional layer within DCGAN. We present the related parameters of the three layers in Table 4.

Benefiting from the reconfigurability of FPGA, we can re-implement the specific accelerator array towards FCN. For the third transposed convolutional layer \(({16, 2, 10, 10, 21, 21})\) in FCN, as its kernel size \(K\) and input width \(I_{W}\) is 16 and 10, we need to apply tiling in output channel owing to the resource limitation. Leveraging the resource model devised above, we can assign the \(O_{TC}\) as 2 without exceeding DSP capacity. We consequently devise the accelerator array \((256, 10, 2)\). Typically, we can load the weights of this layer, totally 0.9 Mb, into on-chip Weight Buffer. By the analytical model, we can assign the \(N_{i}\) and \(N_{o}\) as 200 while satisfying the memory limitation. Then, we iteratively multiply the rows of feature maps with corresponding weights. In this way, we achieve the 4761.64 GOPS performance by costing 5,120 DSPs. For the first transposed convolutional layer \((4, 2, 1, 1, 21, 21)\) in the FCN, we can design the corresponding accelerator array \((16, 1, 10)\) by applying tiling in the output channel dimension. According to the proposed design space exploration method, we can store the 6.9 Kb weights in the chip and set \(N_{i}/N_{o}\) as 1,000 under the memory limitation. Then, we can obtain the 184.20 GOPS by utilizing 160 DSPs. Similarly, for the second transposed convolutional layer \((4, 2, 4, 4, 21, 21)\) of the FCN, we implement the array \((16, 4, 10)\) to execute. We store the 6.9 Kb weights in the on-chip memory and assign the \(N_{i}/N_{o}\) as 500. By implementing the array in the FPGA board, we obtain the 742.40 GOPS performance at the cost of 640 DSPs.

For the target layer \(({9, S, 32, 32, 32, 1})\) of FSRCNN based on the CIFAR-10 dataset, we devise the corresponding accelerator array with 81 rows and 32 columns. As the output channel \(O_{C}\) is 1, we need not apply tiling. The weight volume of this layer is relatively small and we can store these weights in on-chip memory. And we assign the \(N_{i}\) and \(N_{o}\) as 4 while satisfying the memory constrain and computation need. When handling this layer, we set the stride size S as different values, such as 2, 3, and 4. As the experimental results demonstrate, the distinct stride sizes have no obvious effects on overall performance, different from the previous works [5, 12]. This is because we can concurrently undertake Flattening Phase and another phase. And it typically takes up more cycles than Overlapping Phase with kernel size being larger than these stride sizes, hiding the latency of both phases.

On the other hand, we additionally execute the FSRCNN on the Set5 dataset, where the image size can be larger than that of CIFAR-10 dataset. For larger images, we typically adopt tiling in the output channel and input width dimensions. And the adjacent tiling outputs can have \(K-S\) overlapping elements, which should be accumulated to generate the final results. When taking the image of size \(256 \times 256\) as the input, the parameter of the transposed convolutional layer can be \((9, S, 256, 256, 32, 1)\). Instead of applying tiling in the output channel dimension whose size is 1, we can apply tiling in the input width dimension to execute the layer. By assigning \(I_{TW}\) as 32, we can implement the accelerator array of size (81, 32, 1) to execute the transposed convolution tiles. Additionally, as each tile output overlaps with adjacent one by \(K-S\) elements, we just need accumulate these overlapping elements to obtain the final results. Generally, we can store the weights in the chip and set \(N_i/N_o\) as 2. Finally, we can achieve the nearly identical performance as performing the layer in the CIFAR-10 dataset.

When accelerating the FSRCNN in the works [5, 12], both of which transform the transposed convolution into multiple convolution operations in an off-line way, their performance is sensitive to the stride size. The proposed intermediate-centric dataflow can reap steady acceleration performance and avoid the enormous transformation overhead. Furthermore, by avoiding the off-line transformation overhead and frequent communication between host/client devices, the intermediate-centric dataflow is more efficient to accelerate the transposed convolution of Back Propagation while performing on-device training in federated learning application [18]. Generally, the work [12] achieves higher DSP performance, because they adopt the FIR algorithm to perform the transformed convolution kernels, so that the multiplication operations can be reduced see Table 6. However, this work demands iteratively transforming the kernel in the software, which impedes it to be applied in accelerating the Error Propagation of the training process. Moreover, it omits directly the padded zeros surrounding the input feature maps after converting transposed convolution, which can also damage the result quality. The proposed intermediate-centric dataflow can directly perform the backward-stencil computation computation and speed up the transposed convolution in-the-fly without any approximation, while keeping reasonable acceleration performance. On the other hand, by solving the formulated problem of Section 4 in these experiments, we find that \(T_{comp}\) is generally larger than \(T_{load}\), which means that the accelerator array is mainly computation-bounded towards the selected transposed convolutional layers. Consequently, by effectively leveraging the DSP resource with high frequency, we can attain better performance in the accelerator architecture as the computation resources increase.

Furthermore, we discuss the scalability of the intermediate-centric dataflow and the hardware architecture. When we implement the accelerator array in the FPGA board, the scalability can be bounded by several factors: 1) The DSPs are distributed in columns in the FPGA chips. For example, the 6840 DSPs are distributed in 32 columns in the U200 boards. We generally obtain the distorted layout of the square array after placement and routing leveraging the EDA tools due to the DSP distribution in rectangle shape. As the proposed accelerator arrays are designed in rectangle shape, we can still keep the shape almost unchanged after placement and routing, by leveraging the optimization methods as mentioned in the work [25]. 2) The chips in the state-of-the-art FPGA boards generally contain several dies each of which consists of a part of the resources. When the size of accelerator array grows larger, the array can be implemented across the dies, which can limit the frequency. As our proposed dataflow indeed works efficiently, these limitations are mainly caused by the physical design of the FPGA boards. On the other hand, while the Google TPUs [9] contain \(512\,\times \,512\) and \(256\,\times \,256\) matrix array, our dataflow can probably avoid the resource limitation in the ASIC technique.

As we aim at proposing the intermediate-centric dataflow to accelerate the transposed convolution and perform its backward-stencil computation, we directly adopt the commonly used data types as the works [19, 20]. And the possible effects on the accuracy from quantization have been presented in the work [20].