Designing Deep Learning Models on FPGA with Multiple Heterogeneous Engines

CNNs are a category of DNN mostly suited for image-focused tasks such as image classification, segmentation, recognition, and so on. CNNs work by processing input images in the form of matrices with pre-defined dimensions. For a CNN to be able to output a desirable prediction it needs to be trained over several iterations with the help of already classified datasets.

CNNs are made up of several layers, where each one gets its input feature map (IFM) from the previous layer and outputs its resulting FM to the next one. Feature maps are the input and output of each layer, composed of two-dimensional arrays of pre-determined sizes. The number of filters used in a layer will result in the same number of output FMs (OFM). The most common types of layers of a CNN are the convolutional, the pooling, the activation function, the fully connected or dense, and batch normalization. The last one is used to normalize the average and the standard deviation of the layers output FM to zero and one, respectively [21]. This improves overall accuracy and speeds up the training process.

The convolutional layer is the most important component of a CNN responsible for extracting features out of its input FM. This is done by convoluting each channel of the input FM with a predefined kernel. A kernel is a small two-dimensional array composed of weights that are trained to achieve the best possible predictions. A 2D convolution is done by processing a single kernel with its two-dimensional input channel. The kernel is overlaid with the input FM array and the dot-product is performed: The multiplications of each overlapped value are added together and then stored on a position of the output array. A bias value is added to the result, which is also a trained parameter. The process is repeated one position to the right until the output array is filled. There are two parameters that alter the way the convolution is done: the padding and the stride. The stride dictates on how many positions the kernel shifts after each dot-product, and the padding determines how many zero valued positions are added to the edges of the input array. The size of the output FM is calculated with Equation (1) [33], where \(i_h\) and \(i_w\) are the input FM array height and width, the \(k_h\) and \(k_w\) are the kernel height and width, the s is the stride, and the p is the padding.

The number of total multiplications in a 2D convolution can be calculated using Equation (2), where OFM is the output FM and k is the kernel.

For a 3D convolution the output FM is composed of several two-dimensional arrays, or channels, the same number as the number of the filters used. A filter is composed of several channels, comparable to a 3D kernel. As in the 2D convolution, several dot-products are performed with the same channels (e.g., the first filter channel is only computed with the first input FM channel, the second filter channel with the second input FM channel, and so forth) and the results of these dot-products are added together along with the bias and stored in a position of a channel of the OFM. Each filter used will result in a different channel on the OFM.

The number of total multiplications in a 3D convolution can be calculated using Equation (3), where OFM is the output FM, \(N_{filters}\) is the number of filters, and F is the Filter.

Depthwise separable convolutions are a less-expensive way of doing convolutions with a large number of filters [40]. The depthwise separable convolution works by using channel-independent dot-products, much like several 2D convolutions, resulting in an intermediate FM with the same number of channels as the filter used, and then doing a convolution with several filters with 1×1-sized channels with the intermediate FM of the previous convolution. Like the 3D convolution, the output FM has the same number of channels as the number of filters used in the depthwise convolution.

The number of total multiplications in a depthwise separable convolution can be calculated using Equation (4) where OFM is the output FM, IFM is the output FM.

Atrous or dilated convolutions allow the control of the resolution of the convolutions by adding a new parameter, the rate. The rate defines the number of spaces between the values of the kernel. This allows for the convolutions to have a wider field of view with the same kernel size.

The size of this convolution output is calculated with Equation (5) where \(i_h\) and \(i_w\) are the input height and width, the \(k_h\) and \(k_w\) are the kernel height and width, the s is stride, the p is the padding, and r is the rate.

The number of multiplications of an Atrous Convolution is also given by Equation (3).

Pooling layers are used to reduce the size of the input FM, and by doing so they make the detected features more prevalent and reduce memory consumption. The most-used types of pooling layers are the max pooling, which gets the maximum value of a window, and average pooling the averages the elements of the window.

The activation function layer simply passes each value of the input FM through a non-linear function. The most common activation functions are the Sigmoid, Leaky ReLU, and the ReLU.

The number of computations involved in a CNN is usually very high due to the number of Multiply and Accumulate (MAC) operations. As CNN models get deeper, the number of operations and the storage requirements increase.

In a typical FPGA-based CNN accelerator [1] there is a many-core computing structure and several levels of memory hierarchy with on-chip and off-chip memory. These systems are composed of input buffers that fetch the input FMs and the kernel values from the external memory through one or more DMAs. The input FMs are streamed into processing elements (PE), and on-chip memories are used to store data locally. There are also on-chip output buffers to store the output FMs to be transferred back to the external memory. A processor controls the scheduling of operations sending control signals to the communication and processing modules.

To efficiently use FPGAs to run neural networks, the CNNs can be compressed to reduce the use of memory, transfer sizes, and operations, while taking into account the minimization of accuracy loss. Compression of a CNN is mostly done with quantization.

Data can be quantized to any number of levels using fewer bits to reduce the complexity of the hardware and increase the throughput of the system but it may also decrease the overall accuracy of the model. The designer chooses the best tradeoff between accuracy and throughput and/or hardware resource consumption while designing the architecture. A fixed-point number representation is usually chosen due to the less computationally expensive operations associated with it, in comparison to floating point representation.

Quantization can be performed after the model is trained by simply converting the floating points values into the chosen data size, or during training where the weights and activations are converted to the low-cost representation during the forward pass. The latter gives better results due to the implicit fine-tuning.

Two types of processing flow approaches are typically used to run CNNs on FPGAs, one in which one computing engine runs each layer iteratively, and another where the whole model computation is mapped on the FPGA and each layer is computed in sequence in a pipelined fashion. A fully streamed pipeline accelerator implements all layers individually. Image data read from external memory is fully processed through all dedicated hardware units until the final result that is read back to external memory.

While pipelined accelerators might be faster due to the low memory transfers, they are constrained to a specific CNN model and are more hardware-expensive than the all-purpose configurable engine, because all the individual layers must be simultaneously mapped in the hardware.

Two well-known illustrative examples of these two processing flows are Angel-Eye [17], which uses a single computing engine, and MALOC [16], which uses a pipelined layer processor.

The Angel-Eye [17] is a design flow that uses a runtime-configurable accelerator of CNN models. It uses instructions to describe layers and data transfers to be executed by the accelerator. The quantization is done after training, where afterwards each weight is fine-tuned to get the best accuracy possible. By fusing layers [4], several can be processed in the same iteration of the engine, thus decreasing the number of intermediate memory accesses.

MALOC [16] is a pipelined architecture where all intermediate results are passed through each processing element (PE) resulting in less memory transfer. MALOC uses 16-bit fixed-point representations, loop tiling, as well as fused layers to decrease the amount of buffered data. It also prunes dense layers to decrease the total number of weights used without decreasing the accuracy.

Most recent CNN accelerators [6, 18, 20, 22, 28, 47] are based on these two approaches with some optimizations to improve efficiency.

To support multiple different types of layers in a single engine, some works consider optimized matrix multiplication engines or a systolic array, where a regular structure of processing elements [9, 42] is laid out and the data is shifted between adjacent PEs, removing the need of global interconnects. The model is mapped out into a feasible PE array configuration, making use of the exploitation of data reuse. Data converters are used to convert the data input into a matrix multiplication or systolic format that executes each type of layer. Using a general core for layer processing allows the utilization of well-optimized structures but requires data converters that could be expensive.

The utilization of multiple engines for CNN implementation was already proposed in previous works. Any pipelined architecture with full implementation of all layers in the FPGA is a multi-engine architecture. It allows full optimization of all layers but requires high resource count and high memory bandwidth [20]. In Reference [32] multiple convolution engines are used to increase the computational efficiency while running a particular layer. The work only considers engines for convolutional layers. The right balance is achieved with several engines that require large number of resources and memory bandwidth. In Reference [37] the objective is to map multiple CNN models in a single FPGA. In this case, a multiple engine is used with each engine dedicated to run one CNN model. This uses the multiple engine approach for a coarser optimization where each engine is optimized for a particular model. In Reference [44] the authors consider two engines tightly connected, one for 3D convolutions and the other for depthwise convolutions targeting the MobileNet model. The dataflow is optimized for this sequence of layers, but the idea of using a sequence of two engines is to optimize the implementation of each type of convolution. In Reference [9] a single engine with a sequence of layers is proposed. Like the work in Reference [44], a custom single engine is designed with multiple modules to run different layers. An example considers a custom dataflow with a depthwise module followed by convolution. The implementation of each layer type is optimized, but their sequential execution introduces some computational inefficiency. In Reference [8] a dual convolutional engine is used, similar to the work of Reference [32]. An optimization algorithm is used to determine the configuration of the array of processing elements that improves the execution of the model. The work only considers convolutional layers. These works highlight the importance of using multiple engines as a way to improve the computational efficiency of a single engine, and the works from References [9, 44] approach the design of a single engine with a sequence of modules for different layer types.

This article proposes a generalization of the CNN accelerator with multiple engines. An engine is an independent processing unit, with computing cores and memory, that runs in parallel with other engines. Engines are dynamically and statically configurable and can have pre- and post-processing non-trainable layers. Each trainable layer type has a configurable engine designed to run it efficiently. Engines with multiple trainable layers are not supported. Any of the previous works can be designed with the proposed architecture. The work from Reference [32] can be designed with multiple convolutional engines. The work from Reference [37] can be implemented considering engines that support the execution of all CNN models. Additionally, the architecture proposed in this article allows sharing engines among different CNN models that can potentially improve the solution. The work from Reference [44] is implemented with two different engines: one for 3D convolution and the other for depthwise convolution. The difference is that using independent engines guarantees the possibility of using both engines without a control or data dependency.

Custom-designed FPGA accelerators using hardware description languages produce very efficient core units [39]. However, a design methodology based on HDL is a time-consuming process, difficult to redesign for other network models, and the design space is hard to explore.

Convolutional neural networks consist of well-defined regular structures that permit the development of domain-specific tools to map CNN on FPGA and the exploration of the design space. A common approach of many of these tools is to consider high-level synthesis to generate CNN accelerators. With HLS it is possible to generate hardware descriptions from the high-level specification of the algorithm in a programming language, such as C/C++ or OpenCL.

An earlier approach to automatically map a CNN on FPGA was DNNWEAVER [31]. It automatically generates a Verilog description of an accelerator from a neural model and a target FPGA device. The CNN is specified in the training framework Caffe [23], and the Verilog description uses designer-optimized templates. The framework includes an optimization tool to reorganize and batch the model operations to improve the utilization of internal resources of the FPGA.

fpgaConvNet [36] is another framework that automatically maps CNNs onto FPGAs. From a custom high-level description of the model, it maps all layers in a pipelined streaming architecture. It considers convolutional, dense, pooling, and activation layers. DeepBurning [41] is similar to fpgaConvNet but schedules the operations dynamically.

FINN [10] from Xilinx automatically maps neural models onto ZYNQ SoC FPGAs. FINN already considers HLS to describe the mapping of the model into a streaming architecture. The tool is particularly efficient to design low-quantization network models. The computing modules per layer can be customized in terms of processing throughput.

HLS4ml [14] is an open-source software–hardware workflow used to automatically implement DNNs in FPGA or ASIC. The flow supports quantization and pruning, whose optimized result is automatically mapped on the target device. The designer can specify the precision expected for the model and considers performance, latency, and resource utilization metrics to orient the design of the accelerator.

Vitis-AI (https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html, accessed on on 22 January 2023) is a development platform for Xilinx devices used to generate an single engine accelerator for the inference of deep neural network models. The approach uses a specialized configurable hardware core that supports the sequential execution of the most common layers. A set of custom instructions is used to dynamically program the accelerator. A Vitis-AI compiler translates the model into this set of instructions.

A recent compilation flow to map CNNs on FPGA [9] supports some recent layer types, besides the traditional ones, such as dilation and transposed convolution. It includes a versatile systolic array that supports multiple types of layers and dynamic tiling. The compilation flow maps an ONNX (Open Neural Network Exchange) description of the model into HLS and allows a design space exploration of the systolic array. The model is executed sequentially, layer-by-layer. The systolic array only implements the MAC operations and is then followed by a sequence of models for pooling or upsample, concatenation, activation, and batch normalization. The introduction of new layers requires the redesign of the systolic array, and depthwise convolution is introduced as a pre-processing layer before the systolic array. The sequential execution of layers in a single engine introduces some computational inefficiency.

In this work, the workflow for mapping a CNN into an FPGA translates into an partition/ allocation/mapping/scheduling problem. A library of configurable HLS engines was developed with a standard interface wrapper. Given a particular CNN model, the designer has to allocate the necessary engines, partition the CNN layers, and maps them on the allocated engines. While the previous mapping tools consider architectures with a single engine or a full stream of engines, the proposed multi-engine architecture requires mapping a CNN into multiple shared engines.

The network model has to be chosen along with the dataset to train the model and then quantized. The proposed design flow adopts the PyTorch machine learning framework to train the model and Brevitas [29] to quantize.

PyTorch is an open source machine learning framework implemented in Python optimized for both CPUs and GPUs. The training of networks with this framework is done by feeding the training script, where several training parameters are defined such as learning rates, metrics, and loss functions, and a description of the neural network to be used. The network description consists of several layers, where each one is a function with its parameters passed as arguments. These include the number of input channels, number of output channels, kernel size, stride, and padding.

Brevitas is a PyTorch library used for quantization-aware training that implements a set of building blocks at different levels of abstraction to model a reduced precision hardware datapath at training time. This library provides several quantized versions of the standard PyTorch layers that can be replaced with the original PyTorch model. To quantize the model, the original descriptions of the layers are replaced by their quantized versions from Brevitas.

To better integrate PyTorch, Brevitas, and FPGA deployment, three Python-based tools were developed:

Brevitas Converter converts the PyTorch model into a quantized Brevitas network along with the trained weights. It runs through all the layers of the original model and replaces them with the Brevitas version. Afterwards, a study on quantization of data of this network, using Brevitas, has to be done, taking into account the tradeoffs between area and performance from reducing the data bitwidth. This study must be diverse regarding the size in bitwidth of the data used throughout the network for the best bitwidth configuration to be chosen. This work considers bitwidth quantizations of sizes 8, 4, and 2 for both weights and activations. Weights and activations may have different sizes.

The accuracy and computational efficiency of inference is then improved by merging the convolutional layers with the batch normalization layers with a tool developed in Python named Batch Normalization Merger, thus reducing the number of overall layers. Batch Normalization during inference time works like a 1 × 1 convolution, i.e., each FM value is multiplied by the Batch Normalization Weight and added with a Batch Normalization Bias and seen in Equation (6).

When there are no activation function layers (non-linear layers) between convolutional and batch normalization layers, the weight and bias can be fused together to reduce runtime and memory requirements, as shown in Equation (7).where

As seen in the formula above, the four parameters are turned into only two, thus reducing runtime and memory requirements.

The Batch Normalization Merger is a script that iterates every layer and fuses grouped convolutions and batch normalization layers automatically and outputs a new weights file. Due to the nature of quantization, there needs to be further training to reach higher values of accuracy. Fine-tuning is done to improve the accuracy further. The weights are then exported into the FPGA to be used with the developed system using another developed tool in Python named Weights Extractor.

Weights extractor is a Python script that extracts all the weights of a trained Brevitas quantized network into binary files in a format that can be read easily. The weights are stored z-wise in a fixed point format with the weights bitwidth being configurable.

Mapping a model to FPGA consists of partitioning the model into macro layers, allocating a set of engines, mapping the macro layers on the engines, scheduling the execution of the engines, and then generating the accelerator as an IP core to be integrated in the hardware/software system. The whole process is based on a library of engines that run layers. For a layer to be executed in hardware, there must be an engine that supports its execution.

The network model is executed layer-by-layer by sending the corresponding data to the correct engines and receiving the output FMs through the DMAs with the use of the developed drivers. Multiple threads are supported to take advantage of parallel engines, that is, while an engine is waiting for the results of another engine, it can run over a different input sample. This is determined by the scheduling step.

A driver sets the engine parameters through AXI-lite, which can be done by writing to the necessary addresses, starting the engine execution by setting the engine start signal on the control register by AXI-Lite, sending in order the bias and weights through AXI-Stream/DMA from the external memory to the engine in a contiguous fashion, setting up the transfer from the engine output to the external memory through AXI-Stream/DMA, and then sending the FM through AXI-Stream/DMA.

The programmer must develop the main application using the drivers and following the sequence of layers as specified in the graph of macro layers.

ResNet50 is a well-known model for image classification and used as a backbone in many convolutional neural networks. The model was initially quantized with different bitwidths starting from a pre-trained model (see results in Table 2).

It is not the objective of this work to improve the state-of-the-art quantization accuracy with ResNet50. The variation in accuracy among the quantized solutions is just 1.7 pp, except for the most aggressive quantization (\(4\times 2\)), where there is a drop of 10 pp compared to original floating-point model. The accuracies of quantizations \(8\times 2\) and \(4\times 4\) are very close but quantization \(4\times 4\) is slightly better. Comparing the \(4\times 4\) quantization against \(8\times 8,\) there is a drop of 2 pp in accuracy. Since the design targets a low-density FPGA to run a large model, the \(4\times 4\) was chosen. The model design and optimization step was applied to the full model to run batch fusing, fine-tune, and extract the final weights.

The first layer of ResNet50 is a convolution (\(64\times 7 \times 7\) filters) followed by batch normalization, ReLU, and \(3\times 3\) max pooling. Considering the singular characteristics of the layer, a dedicated engine was chosen for this layer. Since the input is just three channels, the depthwise with accumulation engine was used. The last layer does average pool of the whole channels followed by a dense layer with 1,000 kernels. A dense layer engine was used with average pool. The hidden layers are repetitions of the bottleneck structure of ResNet50 (see Table 3).

All layers, including the downsample layer, can be implemented with the 3D convolution engine with ReLU and shortcut addition. Batch normalization was merged with the weights in the previous step.

So, the accelerator includes three engines: one for the first layer, one for the last layer, and one for all hidden layers (see Figure 9).

The three engines work in parallel and can be used in a dataflow streaming to process a sequence of images. The architecture of each engine was configured before scheduling, as specified in the engines in the figure. The configurations were determined based on the execution times of macro-layer. Knowing the number of MACS, parameters and map sizes necessary to run each macro-layer, the ratio between layer operations and peak performance of engines was found together with the expected data transfer times, assuming a data transfer of 16 Bytes/CLK (see Table 4).

The 3D core engine has the best expected computation time. However, the non-overlapping data transfers degrade the total peak execution time.

The hardware/software system with the multi-engine architecture was implemented, tested on the board, and the results compared with the state-of-the-art. The multi-engine runs at 200 MHz (see Table 5).

The multi-engine architecture is better than the other \(4\times 4\) solution in terms of GOPS/kLUT and GOPS/DSP. The FPS of the proposed architecture is smaller but it runs ResNet50 while the other work runs ResNet18. Compared to Reference [25], the main difference in the throughput comes from the limited memory bandwidth and on-chip memory of our FPGA device. It is not evident from the results, but the bottleneck of our solution is data communication.

YOLO and all its versions [15] are one-stage object detectors with a common model topology based on convolutional neural networks. The YOLO detector extracts features using a CNN and returns candidate bounding boxes from those features for three different scales: 52 × 52, 26 × 26, and 13 × 13.

The original CNN model of YOLO has convolutional, shortcut, YOLO, upsample, and route layers. Convolutional layers with stride two replace max-pooling. Batch-normalization is applied to all convolutional layers, and all layers use the Leaky ReLU activation function, except the layers before YOLO layers that use a linear activation function. YOLO is able to detect objects of different sizes using three different scales: 52 × 52 to detect small objects, 26 × 26 to detect medium objects, and 13 × 13 to detect big objects. YOLOv3-Tiny replaces convolutions with a stride of two by convolutions with max-pooling and does not use shortcut layers.

Table 6 details the sequence of layers with regards to the input, output, and kernel sizes, and the activation function used in each convolutional layer of YOLOV3-Tiny.

The first part of the network is composed of a series of convolutional and maxpool layers. The object detection and classification part of the network performs object detection and classification at (\(13 \times 13\)) and (\(26\times 26\)) grid scales. The detection at a lower resolution is obtained by passing the feature extraction output over \(3\times 3\) and \(1\times 1\) convolutional layers and a YOLO layer at the end.

The detection at the higher resolution follows the same procedure but uses FMs from two layers of the network. The second detection uses intermediate results from the feature extraction layers concatenated with upscaled FMs used for the lower resolution detection.

Following the design flow, the model was quantized with 8 bits, achieving a precision score 30.8 mAP50 in the COCO 2017 test dataset (the original model with floating-point has a score of 32.9 mAP50). Lower quantizations introduce large errors and therefore were not considered.

The model consists of 3D convolutional layers with maxpool, without maxpool, and with upsample. Different Activation functions are used: leaky, linear, and sigmoid.

The first layer is a convolution (\(16\times 3 \times 3\) filters) followed by leaky function and max pooling. A dedicated engine was also considered for this layer. Since the input is just three channels, the depthwise with accumulation engine was used.

All the remaining layers are based on a 3D convolutional (see Table 7).

All layers can be implemented with the 3D convolution engine. To use a single convolution engine for all layers, it was necessary to design a configurable activation function with support for both Leaky ReLU and Sigmoid.

So, the accelerator includes two engines: one for the first layer and one for all hidden layers (see Figure 10).

The architecture of each engine was also configured initially as specified in the engines in the figure. Once again, knowing the number of MACS, parameters, and map sizes necessary to run each layer, the ratio between layer operations and peak performance of engines were found together with the expected data transfer times, assuming a data transfer of 16 Bytes/CLK (see Table 8).

The hardware/software system with the multi-engine architecture was implemented, tested on the board, and the results compared with the state-of-the-art. The multi-engine runs at 200 MHz (see Table 9).

Only one implementation of YOLOv3-Tiny on FPGA with a \(8\times 8\) quantization was found. Therefore, the proposed solution was also compared with previous solutions with higher quantizations. Compared to the work from Reference [2] that also uses an 8-bit quantization with the same FPGA, the proposed multi-engine is \(4.3\times\) faster and with higher ratios between performance and area.

Semantic segmentation is the process of assigning a label to each pixel of an input image. This is useful for object detection where several objects can be identified. A common architecture for semantic segmentation is the Encoder-Decoder structure [26, 45]. The Encoder-Decoder structure is a two-stage network where the encoder compresses its input through several convolutions, which captures the semantic information, while the decoder predicts the output from the encoder’s output FM. The encoder is usually a network such as ResNet or Xception that captures the semantic information, through its several convolutions, while the decoder assigns each output pixel to a label using the encoder semantic information.

The semantic segmentation model considered in this work was the DeepLabV3+ [13] with a Modified Aligned Xception backbone. DeepLabV3+ is an architecture that employs the Encoder-Decoder structure. A simple decoder is added to its predecessor DeepLabV3, which makes use of the ASPP (Atrous Spatial Pyramid Pooling). ASPP applies several atrous convolutions with different rates in parallel to compute features in multiple scales. ASPP can help with detecting objects that might be too far away or too close to the camera due to use of atrous convolutions with different rates. Different backbones can be used with DeepLabV3+. Recently, with a modified Xception as a backbone, this model was able to improve the results of its predecessor DeepLabV3.

This network consists of 142 Convolutions and ReLU as the activation function. The network contains modifying layers such as feature map sums, average pooling, and interpolations. Figures 11 and 12 show the Modified Aligned Xception and the DeepLabV3+ Networks, respectively, with all the convolutional layers stated in order of execution with their most important parameters as well.

This network features Separable Convolutions, Atrous Convolutions, Shortcut connections with addition, Average Pooling, Bilinear Interpolations, and Map Concatenations. Each Separable Convolution block is composed of a depthwise convolution, a batch normalization layer, a pointwise convolution, another batch normalization layer, and a ReLU layer. A convolution block is composed of a Convolution, a batch normalization layer, and ReLU Layer, except for the last convolution where there is no ReLU layer. The Network is divided in several parts: the Encoder, which is composed of a backbone network—in this case, the Modified Aligned Xception, but could be replaced with another one—and the ASPP block, which gathers features in multiple scales with the use of atrous convolutions. Next to the encoder is the decoder block, which is responsible for decoding the features gathered by the encoder. As for the Modified Aligned Xception, it is divided into three sections: the Entry flow, which consists of several Separable Convolutions, some with a stride of 2 to reduce the size of the FMs; the Middle Flow, which is repeated 16 times in sequence; and the Exit flow, which contains some convolutions with dilation characteristics.

The DeepLabV3+ with an Xception Backbone was trained with the Corsican Fire dataset using only RGB images sized 300 × 300. The Corsican dataset¹ is composed of a total of 1,135 images of fires acquired in the visual range (i.e., RGB values) and in the near infrared (i.e., single channel image) under various conditions of positioning, weather, vegetation, distance to the fire, and brightness. The dataset was divided randomly into three parts: the first being the training partition, where 682 images are used for training the network; the second being the validation partition, where 226 images are tested after each epoch of training to keep track of the network training process; and the third being the test partition, where 226 images are tested to evaluate the network final mIoU. The training was done with a learning rate of 0.02 and with a Cross-entropy loss function. The mIoU reached for validation and testing are 92.45% and 91.5%, respectively.

After training the network, several quantization attempts with varying weights and activation bitwidths were done to determine the best mIoU-to-hardware-resources compromise achievable. Table 10 shows the results obtained.

The bitwidths were tested from 8 to 3 for both activation and weights, as lowering even more the size of the bitwidths would yield mIoU results below 80%. These results are probably not the best achievable with these bitwidths, since we stopped training after 30 epochs. However, at 30 epochs the loss variation is small, meaning that only a small improvement would be achieved with more training epochs. Decreasing both the weights and activations bitwidths lower the accuracy of the network, and with an activation bitwidth of 3, the accuracy decreases significantly, independently of the weights bitwidth. Since using weights with a bitwidth size that is not a power of 2 wastes memory resources, and as the achieved mIoU of 87.31% for a bitwidth of 4 for both activations and weights is only less than 5% lower than the mIoU achieved with the original PyTorch network, the rest of this work will use this bitwidth configuration.

Afterwards, the Batch normalization layers were merged with their respective convolutional layers using the developed tool Batch Normalization Merger, which decreased the mIoU to 81.21%. After further 50 epochs of fine-tune training, the mIoU reached 92.76% (an even higher value than the original PyTorch mIoU of 91.53%). The reason the quantized network performs better than the non-quantized PyTorch network might be due to the noise associated with the quantization.

The architecture was split into the three different engines: 3D Convolution, Depthwise Convolution, and Dilated Convolution. The 3D Convolution engine also computes the pointwise convolutions.

The 3D convolution includes a pre-processing average pooling and a post-processing with ReLU, concatenation with addition, and interpolation.

The average pooling is able to average a variable-sized map into a 1 × 1 map by reading the input stream values and adding them to the previously read values. Once all the values are read, the division by the total number of pixels of the resulting value of the sums is done. The division is replaced by a multiplication of a value calculated statically (\(\frac{2^{25}}{N}\)) and a bit shift represented by a division by \(2^{25}\) to reduce the hardware and execution time, as shown in Equation (12).

The post-processing of the 3D convolution engine includes a bilinear interpolation, which was not in the original implementation of the engine used for the first two models. Interpolation works by applying linear interpolation in two directions. The linear interpolation formula can be seen in Equation (13), where ×1 and ×2 are the known coordinates values in a 1-dimensional plane, Q1 and Q2 are the values associated with those coordinates, and P is the value to be calculated for the new x.

The Bilinear Interpolation formula, seen in Equation (14), can be deduced from the linear interpolation formula when moving on to a 2-dimensional plane.

Implementing this formula with fixed point numbers requires scaling the position of the known points for there not to be any fractional bits. The scaling factor is related to the size of the bigger map, e.g., if the bigger map is has a size of 10 pixels, then the scaling factor would be 9.

The final division of the interpolation can be replaced by a bit shift by first multiplying a factor that is the result of the division between a power of two and the actual denominator. This factor is calculated at the beginning of the interpolation process, since the value is always the same. The formula can then be rewritten:

Before starting the Bilinear Interpolation, the module stores two lines of the map. Afterwards, values are loaded and the interpolation is performed while loading the next line of values.

The final architecture with the three engines is illustrated in Figure 13.

The three engines work in parallel and can be used in a dataflow streaming to process a sequence of images. The architecture of each engine was configured as specified in the engines in the figure.

The hardware/software system has four DMAs: two DMAs are connected to the 3D convolution engine, one DMA is connected to the Depthwise IP, and the other is connected to the dilation convolution engine.

The system was synthesized and implemented achieving a working frequency of 175 MHz. Table 11 shows the Hardware Utilization of the whole system.

As shown, no hardware limits are hit, the highest resource usage being the BRAMs. The FPGA bitstream was then generated and exported along the with the necessary files to Vitis IDE to be able to run the network.

The network was run in the Ultra96-v2 Board, the 226 test images were processed, and their respective mIoU was evaluated (92.7%). The input images, weights, and truth images were stored, before execution, into the board DDR memory. The input images were saved already normalized, so no pre-processing on them is needed. Every output image was compared with the truth images, calculating their mIoU, confirming that the implementation is correct. Then the execution time was recorded and averaged over the 226 runs to figure out the system’s throughput. A throughput of 1.4 frames per second was observed.

The results were compared with the only work found of DeepLabV3+ on FPGA (see Table 12).

The solutions compared in the table target different FPGAs. The work from Reference [27] considers a high-density FPGA. However, the difference in the utilization of resources is much larger than our multi-engine architecture and has throughput \(5.4\times\) lower.