Going Deeper in Spiking Neural Networks: VGG and Residual Architectures

4.1. Input and Output Representation

The main difference between ANN and SNN operation is the notion of time. While ANN inputs are static, SNNs operate based on dynamic binary spiking inputs as a function of time. The neural nodes also receive and transmit binary spike input signals in SNNs, unlike in ANNs, where the inputs and outputs of the neural nodes are analog values. In this work, we consider a rate-encoded network operation where the average number of spikes transmitted as input to the network over a large enough time window is approximately proportional to the magnitude of the original ANN inputs (pixel intensity in this case). The duration of the time window is dictated by the desired network performance (for instance, classification accuracy) at the output layer of the network. A Poisson event-generation process is used to produce the input spike train to the network. Every time-step of SNN operation is associated with the generation of a random number whose value is compared against the magnitude of the corresponding input. A spike event is triggered if the generated random number is less than the value of the corresponding pixel intensity. This process ensures that the average number of input spikes in the SNN is proportional to the magnitude of the corresponding ANN inputs and is typically used to simulate an SNN for recognition tasks based on datasets for static images (Diehl et al., 2015). Figure 1 depicts a particular timed-snapshot of the input spikes transmitted to the SNN for a particular image from the CIFAR-10 dataset. Note that since we are considering per pixel mean subtracted images, the input layer receives spikes whose rate is proportional to the input magnitude with sign equal to the input sign. However, for subsequent layers all spikes are positive in sign since there are generated by spiking neurons in the network. SNN operation of such networks are “pseudo-simultaneous,” i.e., a particular layer operates immediately on the incoming spikes from the previous layer and does not have to wait for multiple time-steps for information from the previous layer neurons to get accumulated. Given a Poisson-generated spike train being fed to the network, spikes will be produced at the network outputs. Inference is based on the cumulative spike count of neurons at the output layer of the network over a given time-window.

4.2. ANN and SNN Neural Operation

ANN to SNN conversion schemes usually consider Rectified Linear Unit (ReLU) as the ANN activation function. For a neuron receiving inputs x_i through synaptic weights w_i, the ReLU neuron output y is given by,

Although ReLU neurons are typically used in a large number of machine learning tasks at present, the main reason behind their usage for ANN-SNN conversion schemes is that they bear functional equivalence to an Integrate-Fire (IF) Spiking Neuron without any leak and refractory period (Cao et al., 2015; Diehl et al., 2015). Note that this is a particular type of Spiking Neuron model (Izhikevich, 2003). Let us consider the ANN inputs x_i encoded in time as a spike train 𝕏_i(t), where the average value of 𝕏_i(t), E[𝕏_i(t)]∝x_i (for the rate encoding network being considered in this work). The IF Spiking Neuron keeps track of its membrane potential, v_mem, which integrates incoming spikes and generates an output spike whenever the membrane potential cross a particular threshold v_th. The membrane potential is reset to zero at the generation of an output spike. All neurons are reset whenever a spike train corresponding to a new image/pattern in presented. The IF Spiking Neuron dynamics as a function of time-step, t, can be described by the following equation,

Note that the neuron dynamics is independent of the actual magnitude of the time-step. Let us first consider the simple case of a neuron being driven by a single input 𝕏(t) and a positive synaptic weight w. Due to the absence of any leak term in the neural dynamics, it is intuitive to show that the corresponding output spiking rate of the neuron is given by E[Y(t)]∝E[𝕏(t)], with the proportionality factor being dependent on the ratio of w and v_th. In the case when the synaptic weight is negative, the output spiking activity of the IF neuron is zero since the neuron is never able to cross the firing potential v_th, mirroring the functionality of a ReLU. The higher the ratio of the threshold with respect to the weight, the more time is required for the neuron to spike, thereby reducing the neuron spiking rate, E[Y(t)], or equivalently increasing the time-delay for the neuron to generate a spike. A relatively high firing threshold can cause a huge delay for neurons to generate output spikes. For deep architectures, such a delay can quickly accumulate and cause the network to not produce any spiking outputs for relatively long periods of time. On the other hand, a relatively low threshold causes the SNN to lose any ability to distinguish between different magnitudes of the spike inputs being accumulated to the membrane potential (the term $\sum _i{w}_i.{𝕏}_i(t)$ in Equation 2) of the Spiking Neuron, causing it to lose evidence during the membrane potential integration process. This, in turn, results in accuracy degradation of the converted network. Hence, an appropriate choice of the ratio of the neuron threshold to the synaptic weights is essential to ensure minimal loss in classification accuracy during the ANN-SNN conversion process (Diehl et al., 2015). Consequently, most of the research work in this field has been concentrated on outlining appropriate algorithms for threshold-balancing, or equivalently, weight normalizing different layers of a network to achieve near-lossless ANN-SNN conversion.

4.3. Architectural Constraints

5.1. Proposed Algorithm: Spike-Norm

However, the above algorithm leads us to the question: Are ANN activations representative of SNN activations? Let us consider a particular example for the case of maximum activation for a single ReLU. The neuron receives two inputs, namely 0.5 and 1. Let us consider unity synaptic weights in this scenario. Since the maximum ReLU activation is 1.5, the neuron threshold would be set equal to 1.5. However, when this network is converted to the SNN mode, both the inputs would be propagating binary spike signals. The ANN input, equal to 1, would be converted to spikes transmitting at every time-step while the other input would transmit spikes 50% of the duration of a large enough time-window. Hence, the actual summation of spike inputs received by the neuron per time-step would be 2 for a large number of samples, which is higher than the spiking threshold (1.5). Clearly, some information loss would take place due to the lack of this evidence integration.

Driven by this observation, we propose a weight-normalization technique that balances the threshold of each layer by considering the actual operation of the SNN in the loop during the ANN-SNN conversion process. The algorithm normalizes the weights of the network sequentially for each layer. Given a particular trained ANN, the first step is to generate the input Poisson spike train for the network over the training set for a large enough time-window. The Poisson spike train allows us to record the maximum summation of weighted spike-input (the term $\sum _i{w}_i.{𝕏}_i(t)$ in Equation 2, and hereafter referred to maximum SNN activation in this text) that would be received by the first neural layer of the network. In order to minimize the temporal delay of the neuron and simultaneously ensure that the neuron firing threshold is not too low, we weight-normalize the first layer depending on the maximum spike-based input received by the first layer. After the threshold of the first layer is set, we are provided with a representative spike train at the output of the first layer which enables us to generate the input spike-stream for the next layer. The process is continued sequentially for all the layers of the network. The main difference between our proposal and prior work (Diehl et al., 2015) is the fact that the proposed weight-normalization scheme accounts for the actual SNN operation during the conversion process. As we will show in the Results section, this scheme is crucial to ensure near-lossless ANN-SNN conversion for significantly deep architectures and for complex recognition problems. We evaluate our proposal for VGG-16 network (Simonyan and Zisserman, 2014), a standard deep convolutional network architecture which consists of a 16 layer deep network composed of 3 × 3 convolution filters (with intermediate pooling layers to reduce the output map dimensionality with increasing number of maps). The pseudo-code of the algorithm is given below.

6.1. ReLUs at Each Junction Point

As we will show in the Results section, application of our proposed Spike-Norm algorithm on such a residual architecture resulted in a converted SNN that exhibited accuracy degradation in comparison to the original trained ANN. We hypothesize that this degradation is attributed mainly to the absence of any ReLUs at the junction points. Each ReLU when converted to an IF Spiking Neuron imposes a particular amount of characteristic temporal delay (time interval between an incoming spike and the outgoing spike due to evidence integration). Due to the shortcut connections, spike information from the initial layers gets instantaneously propagated to later layers. The unbalanced temporal delay in the two parallel paths of the network can result in distortion of the spike information being propagated through the network. Consequently, as shown in Figure 2B, we include ReLUs at each junction point to provide a temporal balancing effect to the parallel paths (when converted to IF Spiking Neurons). An ideal solution would be to include a ReLU in the parallel path, but that would destroy the advantage of the identity mapping.

6.2. Same Threshold of All Fan-In Layers

As shown in the next section, direct application of our proposed threshold-balancing scheme still resulted in some amount of accuracy loss in comparison to the baseline ANN accuracy. However, note that the junction neuron layer receives inputs from the previous junction neuron layer as well as the non-identity neuron path. Since the output spiking activity of a particular neuron is also dependent on the threshold-balancing factor, all the fan-in neuron layers should be threshold-balanced by the same amount to ensure that input spike information to the next layer is rate-encoded appropriately. However, the spiking threshold of the neuron layer in the non-identity path is dependent on the activity of the neuron layer at the previous junction. An observation of the typical threshold-balancing factors for the network without using this constraint (shown in Figure 2C) reveal that the threshold-balancing factors mostly lie around unity after a few initial layers. This occurs mainly due to the identity mapping. The maximum summation of spike inputs received by the neurons in the junction layers are dominated by the identity mapping (close to unity). From this observation, we heuristically choose both the thresholds of the non-identity ReLU layer and the identity-ReLU layer equal to 1. However, the accuracy is still unable to approach the baseline ANN accuracy, which leads us to the third design constraint.

6.3. Initial Non-residual Pre-processing Layers

An observation of Figure 2C reveals that the threshold-balancing factors of the initial junction neuron layers are significantly higher than unity. This can be a primary reason for the degradation in classification accuracy of the converted SNN. We note that the residual architectures used by authors in He et al. (2016a) use an initial convolution layer with a very wide receptive field (7 × 7 with a stride of 2) on the ImageNet dataset. The main motive behind such an architecture was to show the impact of increasing depth in their residual architectures on the classification accuracy. Inspired by the VGG-architecture, we replace the first 7 × 7 convolutional layer by a series of three 3 × 3 convolutions where the first two layers do not exhibit any shortcut connections. Addition of such initial non-residual pre-processing layers allows us to apply our proposed threshold-balancing scheme in the initial layers while using a unity threshold-balancing factor for the later residual layers. As shown in the Results section, this scheme significantly assists in achieving classification accuracies close to the baseline ANN accuracy since after the initial layers, the maximum neuron activations decay to values close to unity because of the identity mapping.

7.1. Datasets and Implementation

We evaluate our proposals on standard visual object recognition benchmarks, namely the CIFAR-10 and ImageNet datasets. Experiments performed on networks for the CIFAR-10 dataset are trained on the training set images with per-pixel mean subtracted and evaluated on the testing set. We also present results on the much more complex ImageNet 2012 dataset that contains 1.28 million training images and report evaluation (top-1 and top-5 error rates) on the 50,000 validation set. 224 × 224 crops from the input images are used for this experiment.

We use VGG-16 architecture (Simonyan and Zisserman, 2014) for both the datasets. ResNet-20 configuration outlined in He et al. (2016a) is used for the CIFAR-10 dataset while ResNet-34 is used for experiments on the ImageNet dataset. As mentioned previously, we do not utilize any batch-normalization layers. For VGG networks, a dropout layer is used after every ReLU layer except for those layers which are followed by a pooling layer. For Residual networks, we use dropout only for the ReLUs at the non-identity parallel paths but not at the junction layers. We found this crucial for achieving training convergence. Note that we have tested our framework only for the above mentioned architectures and datasets. There is no selection bias in the reported results.

Our implementation is derived from the Facebook ResNet implementation code for CIFAR and ImageNet datasets available publicly¹. We use same image pre-processing steps and scale and aspect-ratio augmentation techniques as used in² (for instance, random crop, horizontal flip and color normalization transforms for the CIFAR-10 dataset). We report single-crop testing results while the error rates can be further reduced with 10-crop testing (Krizhevsky et al., 2012). Networks used for the CIFAR-10 dataset are trained on 2 GPUs with a batchsize of 256 for 200 epochs, while ImageNet training is performed on 8 GPUs for 100 epochs with a similar batchsize. The initial learning rate is 0.05. The learning rate is divided by 10 twice, at 81 and 122 epochs for CIFAR-10 dataset and at 30 and 60 epochs for ImageNet dataset. A weight decay of 0.0001 and a momentum of 0.9 is used for all the experiments. Proper weight initialization is crucial to achieve convergence in such deep networks without batch-normalization. For a non-residual convolutional layer (for both VGG and ResNet architectures) having kernel size k × k with n output channels, the weights are initialized from a normal distribution and standard deviation $\sqrt{\frac{2}{k^{2}n}}$. However, for residual convolutional layers, the standard deviation used for the normal distribution was $\frac{\sqrt{2}}{k^{2}n}$. We observed this to be important for achieving training convergence and a similar observation was also outlined in Hardt and Ma (2016) although their networks were trained without both dropout and batch-normalization.

7.2. Experiments for VGG Architectures

Our VGG-16 model architecture follows the implementation outlined in³ except that we do not utilize the batch-normalization layers. We used a randomly chosen mini-batch of size 256 from the training set for the weight-normalization process on the CIFAR-10 dataset. While the entire training set can be used for the weight-normalization process, using a representative subset did not impact the results. We confirmed this by running multiple independent runs for both the CIFAR and ImageNet datasets. The standard deviation of the final classification error rate after 2,500 time-steps was ~ 0.01%. All results reported in this section represent the average of 5 independent runs of the spiking network (since the input to the network is a random process). No notable difference in the classification error rate was observed at the end of 2, 500 time-steps and the network outputs converged to deterministic values despite being driven by stochastic inputs. For the SNN model based weight-normalization scheme (Spike-Norm algorithm) we used 2,500 time-steps for each layer sequentially to normalize the weights.

Table 1 summarizes our results for the CIFAR-10 dataset. The baseline ANN error rate on the testing set was 8.3%. Since the main contribution of this work is to minimize the loss in accuracy during conversion from ANN to SNN for deep-layered networks and not in pushing state-of-the-art results in ANN training, we did not perform any hyper-parameter optimization. However, note that despite several architectural constraints being present in our ANN architecture, we are able to train deep networks that provide competitive classification accuracies using the training mechanisms described in the previous subsection. Further reduction in the baseline ANN error rate is possible by appropriately tuning the learning parameters. For the VGG-16 architecture, our implementation of the ANN-model based weight-normalization technique, proposed by Diehl et al. (2015), yielded an average SNN error rate of 8.54% leading to an error increment of 0.24%. The error increment was minimized to 0.15% on applying our proposed Spike-Norm algorithm. Note that we consider a strict model-based weight-normalization scheme to isolate the impact of considering the effect of an ANN vs. our SNN model for threshold-balancing. Further optimizations of considering the maximum synaptic weight during the weight-normalization process (Diehl et al., 2015) is still possible.

Previous works have mainly focused on much shallower convolutional neural network architectures. Although Hunsberger and Eliasmith (2016) reports results with an accuracy loss of 0.18%, their baseline ANN suffers from some amount of accuracy degradation since their networks are trained with noise (in addition to architectural constraints mentioned before) to account for neuronal response variability due to incoming spike trains (Hunsberger and Eliasmith, 2016). It is also unclear whether the training mechanism with noise would scale up to deeper layered networks. Our work reports the best performance of a Spiking Neural Network on the CIFAR-10 dataset till date.

The impact of our proposed algorithm is much more apparent on the more complex ImageNet dataset. The rates for the top-1 (top-5) error on the ImageNet validation set are summarized in Table 2. Note that these are single-crop results. The accuracy loss during the ANN-SNN conversion process is minimized by a margin of 0.57% by considering SNN-model based weight-normalization scheme. It is therefore expected that our proposed Spike-Norm algorithm would significantly perform better than an ANN-model based conversion scheme as the pattern recognition problem becomes more complex since it accounts for the actual SNN operation during the conversion process. Note that Hunsberger and Eliasmith (2016) reports a performance of 48.2%(23.8%) on the first 3072-image test batch of the ImageNet 2012 dataset.

At the time we developed this work, we were unaware of a parallel effort to scale up the performance of SNNs to deeper networks and large-scale machine learning tasks. The work was recently published in Rueckauer et al. (2017). However, their work differs from our approach in the following aspects: (i) Their work improves on prior approach outlined in Diehl et al. (2015) by proposing conversion methods for removing the constraints involved in ANN training (discussed in section 4.3). We are improving on prior art by scaling up the methodology outlined in Diehl et al. (2015) for ANN-SNN conversion by including the constraints. (ii) We are demonstrating that considering SNN operation in the conversion process helps to minimize the conversion loss. Rueckauer et al. (2017) uses ANN based normalization scheme used in Diehl et al. (2015). While removing the constraints in ANN training allows authors in Rueckauer et al. (2017) to train ANNs with better accuracy, they suffer significant accuracy loss in the conversion process. This occurs due to a non-optimal ratio of biases/batch-normalization factors and weights Rueckauer et al. (2017). This is the primary reason for our exploration of ANN-SNN conversion without bias and batch-normalization. For instance, their best performing network on CIFAR-10 dataset incurs a conversion loss of 1.06% in contrast to 0.15% reported by our proposal for a much deeper network. The accuracy loss is much larger for their VGG-16 network on the ImageNet dataset—14.28% in contrast to 0.56% for our proposal. Although Rueckauer et al. (2017) reports a top-1 SNN error rate of 25.40% for an Inception-V3 network, their ANN is trained with an error rate of 23.88%. The resulting conversion loss is 1.52% and much higher than our proposals. The Inception-V3 network conversion was also optimized by a voltage clamping method, that was found to be specific for the Inception network and did not apply to the VGG network Rueckauer et al. (2017). Note that the results reported on ImageNet in Rueckauer et al. (2017) are on a subset of 1, 382 image samples for Inception-V3 network and 2,570 samples for VGG-16 network. Hence, the performance on the entire dataset is unclear. Our contribution lies in the fact that we are demonstrating ANNs can be trained with the above-mentioned constraints with competitive accuracies on large-scale tasks and converted to SNNs in a near-lossless manner.

This is the first work that reports competitive performance of a Spiking Neural Network on the entire 50,000 ImageNet 2012 validation set.

7.3. Experiments for Residual Architectures

Our residual networks for CIFAR-10 and ImageNet datasets follow the implementation in He et al. (2016a). We first attempt to explain our design choices for ResNets by sequentially imposing each constraint on the network and showing their corresponding impact on network performance in Figure 3. The “Basic Architecture” involves a residual network without any junction ReLUs. “Constraint 1” involves junction ReLUs without having equal spiking thresholds for all fan-in neural layers. “Constraint 2” imposes an equal threshold of unity for all the layers while “Constraint 3” performs best with two pre-processing plain convolutional layers (3 × 3) at the beginning of the network. The baseline ANN ResNet-20 was trained with an error of 10.9% on the CIFAR-10 dataset. Note that although we are using terminology consistent with He et al. (2016a) for the network architectures, our ResNets contain two extra plain pre-processing layers. The converted SNN according to our proposal yielded a classification error rate of 12.54%. Weight-normalizing the initial two layers using the ANN-model based weight-normalization scheme produced an average error of 12.87%, further validating the efficiency of our weight-normalization technique.

On the ImageNet dataset, we use the deeper ResNet-34 model outlined in He et al. (2016a). The initial 7 × 7 convolutional layer is replaced by three 3 × 3 convolutional layers where the initial two layers are non-residual plain units. The baseline ANN is trained with an error of 29.31% while the converted SNN error is 34.53% at the end of 2,500 timesteps. The results are summarized in Table 3 and convergence plots for all our networks are provided in Figure 4. It is worth noting here that the main motivation of exploring Residual Networks is to go deeper in Spiking Neural Networks. We explore relatively simple ResNet architectures, as the ones used in He et al. (2016a), which have an order of magnitude fewer parameters than standard VGG-architectures. Further hyper-parameter optimizations or more complex architectures are still possible. While the accuracy loss in the ANN-SNN conversion process is more for ResNets than plain convolutional architectures, yet further optimizations like including more pre-processing initial layers or better threshold-balancing schemes for the residual units can still be explored. This work serves as the first work to explore ANN-SNN conversion schemes for Residual Networks and attempts to highlight important design constraints required for minimal loss in the conversion process.

7.4. Computation Reduction Due to Sparse Neural Events

ANN operation for prediction of the output class of a particular input requires a single feed-forward pass per image. For SNN operation, the network has to be evaluated over a number of time-steps. However, specialized hardware that accounts for the event-driven neural operation and “computes only when required” can potentially exploit such alternative mechanisms of network operation. For instance, Figure 5 represents the average total number of output spikes produced by neurons in VGG and ResNet architectures as a function of the layer for ImageNet dataset. A randomly chosen minibatch was used for the averaging process. We used 500 timesteps for accumulating the spike-counts for VGG networks while 2,000 time-steps were used for ResNet architectures. This is in accordance to the convergence plots shown in Figure 4. An important insight obtained from Figure 5 is the fact that neuron spiking activity becomes sparser as the network depth increases. Hence, benefits from event-driven hardware is expected to increase as the network depth increases. While an estimate of the actual energy consumption reduction for SNN mode of operation is outside the scope of this current work, we provide an intuitive insight by providing the number of computations per synaptic operation being performed in the ANN vs. the SNN.

The number of synaptic operations per layer of the network can be easily estimated for an ANN from the architecture for the convolutional and linear layers. For the ANN, a multiply-accumulate (MAC) computation takes place per synaptic operation. On the other hand, a specialized SNN hardware would perform an accumulate computation (AC) per synaptic operation only upon the receipt of an incoming spike. Hence, the total number of AC operations occurring in the SNN would be represented by the dot-product of the average cumulative neural spike count for a particular layer and the corresponding number of synaptic operations. Calculation of this metric reveal that for the VGG network, the ratio of SNN AC operations to ANN MAC operations is 1.975 while the ratio is 2.4 for the ResNet (the metric includes only ReLU/IF spiking neuron activations in the network). However, note the fact that a MAC operation involves an order of magnitude more energy consumption than an AC operation Han et al. (2015b). Hence, the energy consumption reduction for our SNN implementation is expected to be significantly lower in comparison to the original ANN implementation. It is worth noting here that the real metric governing the energy requirement of SNN vs. ANN is the number of spikes per neuron. Energy benefits are obtained only when the average number of spikes per neuron over the inference timing window is <1 (since in the SNN the synaptic operation is conditional based on spike generation by neurons in the previous layer). Hence, to get benefits for energy reductions in SNNs, one should target deeper networks due to neuron spiking sparsity.

While the SNN operation requires a number of time-steps in contrast to a single feed-forward pass for an ANN, the actual time required to implement a single time-step of the SNN in a neuromorphic architecture might be significantly lower than a feedforward pass for an ANN implementation (due to event-driven hardware operation). An exact estimate of the delay comparison is outside the scope of this article. Nevertheless, despite the delay overhead, as highlighted above, the power benefits of event-driven SNNs can significantly increase the energy (power x delay) efficiency of deep SNNs in contrast to ANNs.

Conflict of Interest Statement

AS was employed by Facebook through a summer internship. YY, RW, and CL were employed by company Facebook. The remaining author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.