SaARSP: An Architecture for Systolic-Array Acceleration of Recurrent Spiking Neural Networks

To mimic biological neurons in the nervous system, a spiking neuron model, such as the most prevailing leaky integrate-and-fire (LIF) model [21] is adopted, as shown in Figure 1(b). Common to literally all spiking neural models, operations in one spiking neuron comprise three main steps at each time point \(t\): (1) integration of pre-synaptic spike inputs, (2) update of the post-synaptic membrane potential, and (3) conditional generation of post-synaptic spike output (action potential). As shown in Figure 1(b) and during Step (1), if a particular pre-synaptic neuron fires, the induced pre-synaptic current will be integrated by the post-synaptic neuron. From a modeling perspective, in this case, the corresponding synaptic weight between the two neurons, or more generally a quantity determined by the weight, will be added (accumulated) to the post-synaptic membrane potential. After integrating all pre-synaptic currents, in Step (2), the post-synaptic spiking neuron updates its membrane potential by adding to it the sum of integrated synaptic currents. Temporal decaying of the membrane potential can also be included if the neural model is leaky. In the last step, the spiking neuron compares its updated membrane potential with a pre-determined firing threshold and generates an output spike (action potential) if the membrane potential exceeds the threshold, as shown in Figure 1(b). The same process repeats for all time points involved in a given spike-based task.

Spiking neurons are wired to form a network. The aforementioned temporal processing of individual spiking neurons are brought into a network setting in which neurons communicate with each other and perform computation by receiving and generating stereotyped all-or-none spikes both spatially and temporally. This article considers the most general (deep) multi-layer recurrent spiking neural network (R-SNN) architecture, which comprises multiple feedforward or recurrent layers with inter-layer feedforward connections. The proposed accelerator architecture intends to accelerate SNN inference on a layer-by-layer basis.

While R-SNNs produce complex network dynamics via recurrent connections, they have gathered significant recent research interests that perceive R-SNNs as a promising biologically inspired paradigm for time series data processing, speech recognition, and predictive learning. In particular, the recurrent connections in an R-SNN form temporally based local memory, opening up opportunities for supporting broad ranges of spatiotemporal learning tasks.

Recent advances in R-SNN network architectures and end-to-end supervised training methods have led to high-performance R-SNNs that are not attainable using unsupervised biologically plausible learning mechanisms such as spike-timing dependent plasticity (STDP). For example, deep R-SNNs have been trained using recent SNN backpropagation techniques [4, 56, 57], achieving state-of-the-art accuracy on commonly used spike-based neuromorphic image and speech recognition datasets such as MNIST [31], Fashion-MNIST [55], N-TIDIGITS [3], TI46 speech corpus [33], and Sequential MNIST and TIMIT [20]. Belec et al. [4] presented a powerful R-SNN architecture, called long short-memory spiking neural networks (LSNNs), comprising multiple distinct leaky integrate-and-fire (LIF) recurrent spiking neural populations. Promising supervised and reinforcement learning, and Learning-to-Learn (L2L) capabilities have been demonstrated using LSNNs.

While SNNs have gathered siginificant research interest as promising bio-inspired models of computation, only a very few works have been done on SNN hardware accelerators [15, 24, 38, 39, 45], particularly array-based accelerators [1, 6, 23, 49, 52] due to the fact that the spatiotemporal nature of the spikes make it difficult to design an efficient architecture. Importantly, these limited existing works have primarily focused on feedforward SNNs where there exist very limited works that are capable of executing R-SNNs [2, 17, 37]. For example, [1, 23, 49, 52] introduced a systolic-array-based accelerator for spiking-CNNs. However, these works are only targeting feedforward networks where efficient method to handle recurrence, which produces tightly-coupled data dependency in both time and space, has not been proposed. There is a key difficulty in developing optimized hardware architecture as strict spatiotemporal dependency resides in R-SNNs. Furthermore, RNN accelerator designs and techniques are incompatible with R-SNNs due to unique properties of spiking models, such as disparity in data representation which lead to different trade-offs in data sharing and storage compared to non-spiking models.

There exist few types of neuromorphic hardware that are capable of executing R-SNNs [2, 17, 37]. Akopyan et al. [2] and Davies et al. [17] are the two best-known industrial large-scale neuromorphic chips, based on a many core architecture. Both chips are fully programmable and have capability of executing R-SNNs, as Akopyan et al. [2] is based on intra-core crossbar memory and long-range connections through an inter-core network and Davies et al. [17] adopt neuron-to-neuron mesh routing model [9, 37] presented a multicore neuromorphic processor chip that employs hybrid analog/digital circuit. With novel architecture that incorporates distributed and heterogeneous memory structures with a flexible routing scheme, [37] can support a wide range of networks including recurrent network.

However, existing architectures are limited to process R-SNNs in a time-sequential manner, which requires alternating access to two different types of weight matrices for every time point, i.e., feedforward weight matrix and recurrent weight matrix. The major shortcomings in the above architectures originate from the stereotype that, both the feedforward and recurrent inputs must be accumulated to generate final activations at a given time point, which are the recurrent inputs to the next time point, before the next time point can be processed. For example, large-scale multicore neuromorphic chips, IBM’s TrueNorth with 256M synapses [2] and Intel’s Loihi with 130M synapses [17], are based on the assumption that all weights of the network are fully stored on-chip. Using a very large core memory can reduce inefficiencies, i.e., lack of parallelism and weight reuse, which makes the weight reuse and data movement less important compared to many other practical cases.

On the other hand, the main idea of our article allows parallel compute over multiple time points as opposed to existing architectures and maximizes weight reuse benefits. Our main target is to minimize data-movement energy cost for memory-intensive SNN accelerators, especially for a practical case that the accelerator cannot load entire weight matrices. Without the proposed idea to minimize the data movement, processing R-SNNs require alternating access to two different types of weight matrices for every time point. Based on the fact above, we defined the baseline architecture (without decoupling) which keeps the essential ideas of existing SNN accelerator works and extended them for recurrent SNNs. With inherent advantages in exploiting spatiotemporal parallelism as will discussed in Section 3.2, our main accelerator is based on systolic array architecture. This work is motivated by the lack of optimized accelerator architectures for general recurrent spiking neural networks (R-SNNs).

As discussed in Section 2.2.2, recurrence in R-SNNs introduces tightly coupled data dependencies both in space and time, which may prevent direct parallelization in the time domain and hence limit the overall performance. We address this challenge by proposing a parallel processing of spike input integration by decoupling the integration of feedforward and recurrent synaptic inputs. The key idea here is to re-structure the spike integration process, as shown in Figure 3(b). One key observation is that while the complete processing of a recurrent layer involves temporal data dependency, the feedforward synaptic input integration, i.e., Step 1 corresponding to (1), has no temporal data dependency and can be parallelized over multiple time points. And then, the following steps of recurrent input integration (Step 1*), membrane potential update (Step 2), and spike output generation (Step 3) are done in a sequential manner time-step by time-step. For example, in conventinal approaches, spike integration step at a given time point \(t_{k}\) requires two different weight matrix, which is repeated for every time point, as depicted in Figure 3(a). However, decoupling feedforward and recurrent stage enables to reuse each of the two weight data for consecutive time points in a row, as shown in Figure 3(b). This decoupling scheme can be represented based on two macro-steps below:

Step A: Feedforward spike input integration for \(t_{k}\) to \(t_{k+TW-1}\) over a time window of \(TW\) points.Step B: Process Step 1*, 2, and 3 for \(t_{k}\) to \(t_{k+TW-1}\) sequentially.

In the above, \(TW\) is the time window size which specifies the temporal granularity of the decoupling. For instance, feedforward synaptic integration step is processed first, over \(TW\) time points, followed by the rest steps in sequential manner. Processing the feedforward input integration over multiple time points as in (6) is possible since the output spikes from the preceding layer over the same time window, which are the inputs to the present layer, have already been computed at this point in the layer-by-layer processing sequence. We further introduce the optimization technique in terms of time window size in the later section. Weight data reuse opportunity. Importantly, this decoupling scheme opens up two weight matrix data reuse opportunities. First all, it is easy to see that the feedforward weight matrix \({\bf W}^{F,l}\) can be reused across all \(TW\) time points in Step A. We group the rest of the steps (Step 1*, 2, and 3) into Step B and process it sequentially. This is because that the layer’s spike outputs at the present time point cannot be determined without knowing the spike outputs at the preceding time point, which feed back to the same recurrent layer via recurrent connections according to (2), (3), (4), and (5). Nevertheless, it is important to note that despite the fact that Step B is performed sequentially, decoupling it from Step A allows for reuse of recurrent weight matrix \({\bf W}^{R,l}\) across \(TW\) time points in Step B. The decoupling scheme offers a unifying solution to enhance weight data reuse for both feedforward integration and recurrent integration, and are applicable to both feedforward and recurrent SNNs.

Without the proposed time-domain parallelism, accelerating recurrent layers in the conventional approach effectively operates on a 1-D array that performs serial processing time point by time point, as shown in Figure 3(a). As discussed above, there exist very limited works for SNN hardware accelerators, and most of prior works have primarily focused on feedforward SNNs where the decoupling scheme has not been explored. We keep the essential ideas of existing SNN accelerator works while extending them for recurrent SNNs without time-domain parallelism, giving rise to the baseline architecture adopted throughout this article for comparison.

In contrast, Figure 4 shows the overall SaARSP architecture, comprising controllers, caches, and a reconfigurable systolic array. Memory hierarchy design is critical to the overall performance of neural network acceleration due to its memory-intensive nature. As a standard practice, we adopt three levels of memory hierarchy, as shown in Figure 4: (1) DRAM, (2) Global buffer, and (3) double-buffered L1 cache [30, 46]. We employ programmable data links with simple control logic among the PEs in the 2-D systolic array such that it can be reconfigured to a 1-D array. Each processing element (PE) consists of a scratchpad and an AC unit that accumulates the pre-synaptic weights when the corresponding input spikes are presented. The SaARSP architecture supports two different stationary dataflows based on systolic array.

Systolic Arrays. A major benefit in using systolic arrays is parallel computing in a simultaneous row-wise and column-wise manner with local communications [52]. Furthermore, systolic arrays are inherently suitable for exploiting spatiotemporal parallelism, i.e., across different neurons (space) and across multiple time-points (time). Especially, systolic arrays are well suited for our main idea to perform parallel computing across time-domain. In addition, separately, the row-wise and the column-wise unidirectional links can be adopted and optimized for the specific data disparity in SNNs.

Stationary Dataflows. For non-spiking ANNs, many previous works have proposed to leverage stationary dataflows to reduce expensive data movement [12, 13, 46]. By keeping one type of data (input, weight, or output) in each PE, an input stationary (IS), weight stationary (WS), and output stationary (OS) dataflow reduces the movement of the corresponding data type. However, a stationary dataflow may result in a different impact for spiking models, considering their unique properties. We explore OS and WS flows to mitigate the movements of large volumes of multi-bit Psum and weight data, respectively. We do not consider input stationary (IS) dataflows that are commonly used in conventional DNN accelerators because binary input spikes are of low volume, and reusing binary input data offers limited benefits.

Processing across \(TW\) time points within the chosen time window as in (6) with the decoupling scheme allows the exploitation of temporal parallelism. However, there exist a fundamental trade-off between weight reuse and Psum storage. The key advantage of decoupling, i.e., weight data reuse across time points, may be completely offset by the need for storing incomplete partial results across multiple time points [48]. Blindly decoupling can exacerbate the above trade-off, and even result in worse performance, as shown later in Figures 7 and 8.

In order to address above issue, we introduce a novel time window size optimization (TWSO) technique to address the fundamental trade-off, optimize the window size \(TW\) to maximize the latency and energy dissipation benefits. We present the first work to explore temporal-granularity in terms of TWSO, which is much powerful and flexible than solely applying the decoupling scheme [48].

TWSO Address the Fundamental Tradeoff. As discussed above, the number of time steps that can be executed simultaneously in one array iteration is limited by the array width \(H\). To accommodate higher degrees of time domain parallelisms, we define K as the time-iteration factor such that \(TW = K \cdot H\), i.e., the parallel integration of feedforward pre-synaptic inputs of a recurrent layer consumes \(K\) array processing iterations, as shown in Figure 5. For a given K, the array reuses a single weight matrix, either \({{\bf W}^F}\) or \({{\bf W}^R}\), for \(TW\) time points.

On the other hand, there may exist optimal choices for the value of \(TW\) (\(K\)). According to Figure 5 and Equation (6), the decoupling scheme batches the feedforward and recurrent input integration steps, and hence it enables reuse of both the feedforward and recurrent weight matrices \({{\bf W}^F}\) and \({{\bf W}^R}\) over multiple time points, avoiding expensive alternating access to them across Step A and Step B. However, parallel processing in the time domain can degrade the performance due to an increased amount of partial sums (Psums). Upon completion of Step A across many time points, there exists a large amount of incomplete, multi-bit partial sum (Psum) data waiting to be processed in Step B. More Psums can result in degraded performance due to the increased latency and energy dissipation of Psum data movement across the memory hierarchy. TWSO address the aforementioned fundamental tradeoff by exploring granularity with varying time-window size, and finding optimal point for the tradeoff. TWSO is generally applicable to both non-spiking RNNs and spiking RNNs.

TWSO Offers Application Flexibility. Typically, a decoupling scheme has a key limitation since it does not provide benefit beyond the first layer due to temporal dependency. This is because as the RNN is unrolled over all time points, both the feedforward and recurrent inputs must be accumulated to generate a recurrent layer’s final activations, which are part of the inputs to the next layer, before the next layer can be processed. However, TWSO subdivides all time points into multiple time windows and performs decoupled accumulation of feedforward/recurrent inputs with a granularity specified by time window size, allowing overlapping the processing of different recurrent layers across different time windows. For example, upon completing processing time window i for recurrent layer k, the activities of layer k in time window (i + 1) and those of layer (k + 1) in time window i can be processed concurrently. Therefore, with TWSO, decoupling can be applied to multiple recurrent layers concurrently.

TWSO Is Specifically Beneficial for Spiking Models. Since the partial sums of each layer are multi-bit while its outputs are only binary, the benefit of increased weight data reuse resulted from decoupling may be more easily offset by the increased storage requirement for multi-bit Psums. Optimizing the granularity of decoupling becomes even more critical for R-SNNs as addressed by TWSO.

Also, TWSO alleviates bottleneck due to data bandwidth by allowing weight reuse across chosen time window size as opposed to iterative weight access, and thus the time window size is not enforced due to the memory bandwidth compared to conventional approaches.

As we demonstrate in our experimental studies in Section 5, TWSO can significantly improve the overall performance.

With given inputs described above, the simulator generates addresses for each type of data with respect to the inputs/outputs required to be fed on the systolic array. The simulator estimates memory access and latency based on cycle-accurate read/write traces for each cache level. As the most widely used method, we follow the estimation method presented in [30, 46].

We specifically consider the systolic array specifications in Table 2 for the proposed SaARSP architecture in our experimental studies in Section 5.

Critically, time windows size (TW) specifies the degree of time-domain parallelism of the SaARSP architecture, i.e., the number of simultaneously processed time points during layer computation as in (6). As discussed in Section 3.3, for a systolic-array with with \({\bf H}\), we define K as the time-iteration factor for more clear illustrative purpose, such that \(TW = K \cdot H\). In other words, the parallel integration of feedforward pre-synaptic inputs of a recurrent layer consumes a multiple of \(K\) array processing iterations. We evaluate the proposed SaARSP as \(K\) varies widely from 1 to 64.

For comparison purposes, we consider a baseline architecture which lacks the proposed decoupled feedforward/recurrent synaptic integration approach (Section 3.1) and hence processes each layer in a time-sequential manner, e.g., time step by time step. As aforementioned in Section 3.2 III-B, our baseline well represent the existing feedforward spiking hardware accelerators, which lacks temporal parallelism and decoupling scheme. For fair comparison, we reconfigure the systolic array into a 1-D array with an equal number of PEs such as the same amount of hardware resources are utilized for parallel compute in space.

The proposed SaARSP architecture can accelerate feedforward layers as a simpler special case (Section 4.4.2). Under the output stationary dataflow, Figure 7 shows the normalized latency and energy of processing feedforward layers of 400 spiking neurons with different time window sizes in two different network configurations with 78 and 784 neurons in the preceding layers, respectively. The inter-layer connectivity factor \(C_{H-R}\) is 1.0 for all layers (Section 4.4.1). Two recurrent layers with intra-layer connectivity factor \(C_{R-R}\) set to 0.3 are also included for comparison purposes. The recurrent layers consume greater latency and energy than the feedforward counterparts due to the additional computation caused by recurrent connections. Nevertheless, processing of feedforward connections contributes a considerable overhead. The latency and energy of the feedforward layers with more connections shown in Figure 7(b) can be significantly reduced by increasing the time window size, i.e., the \(K\) value, due to the fact that larger time window sizes offer more weight data reuse opportunities for weight reuse. That is, reusing the same weights for processing larger number of time points mitigates expensive data movement from higher- to lower-level caches. However, it shall be noted that merely increasing time window size may degrade the overall performance, which is reflected in Figure 7(a). These feedforward layers incur reduced weight access overhead due to their fewer connections. On the other hand, employing a greater time window size produces more multi-bit Psum data that must be kept over a larger number of time points, leading to a degradation in overall performance.

Before presenting a more complete evaluation of recurrent layer acceleration in Section 5.3, we first analyze latency and energy dissipation tradeoffs of accelerating Type-1 recurrent layers with varying time window size under the output stationary dataflow.

We investigate how the proposed decoupled feedforward/recurrent input integration, time window size, and stationary dataflow can be jointly applied/optimized for a given network structure based on the eight R-SNN layer benchmarks (B1 to B8) of two different types of Section 4.4.2. We compare the SaARSP architecture with baseline array, which has the same number of PEs but is unable to explore the proposed time-domain parallelism. The results in Figures 10 and 11 are normalized to this baseline.