AIgean: An Open Framework for Deploying Machine Learning on Heterogeneous Clusters

The ML computing stack is shown in Figure 1. At the top of the stack, we have a wide range of Applications & Algorithms, many of these applications having strict performance constraints. At the Cluster Deployment & Communication layer of the stack, we have petabytes of data being transferred, and at the Hardware layer, we have many mathematical operations (typically matrix/vector multiplications) implemented on a computing substrate ranging from programmable processors to custom hardware. Each layer of this stack provides its challenges. For example, at the Applications layer, the user has to decide which error rates are acceptable for their given application. At the Communication layer, the user has to decide how they will connect their devices (consisting of computing devices as well as sensors gathering data). Finally, at the low-level Hardware layer, the user may want to make optimizations on bit-level operations for their given application or define different levels of parallelism. There are opportunities for research at all levels of this stack.

We define software frameworks as those that mainly target CPUs and GPUs that are programmed via software. Leading software ML frameworks include TensorFlow [9], Torch [10], and Caffe [11]. They provide the users with libraries in various programming languages (e.g., Python, C++) to describe their ML applications. These frameworks then compile the applications into a series of instructions to be executed. Furthermore, they offer an interface to create custom layers that can be compiled into instructions to run on different back-end devices. Finally, they also support connectivity across multiple devices. For example, TensorFlow provides an API [12] to run on distributed clusters, where the communication between different devices (CPUs, GPUs, and TPUs [13]) is either through the CPU network link, through NVLink [14] (i.e., a proprietary link between certain NVIDIA GPUs), or via a direct network link. NVIDIA also provides the NVIDIA Collective Communication Library [15], which implements multi-GPU and multi-node communication primitives optimized for NVIDIA GPUs and networking. This enables scaling GPU computations across large numbers of GPUs available on a network and is supported by several popular deep learning frameworks. Note that in the current implementation of the NVIDIA Collective Communication Library, the GPUs do not have direct connections to the network, unlike what we are able to do with FPGAs. The GPUs connect to the server’s network interface through PCIe. We expect that with the acquisition of Mellanox by NVIDIA [16], GPUs will soon also be able to access the network directly and bypass the need for a PCIe transfer.

These frameworks have a high level of customization at the application level. They also allow the users to input custom instructions, but the underlying hardware circuitry is limited to CPU, GPU, and TPU computation and cannot be modified.

Frameworks such as the Xilinx ML Suite and the Intel Deep Learning Accelerator (DLA) provide overlays implemented on FPGAs [17, 18]. An overlay is essentially a programmable engine implemented on the FPGA, which has a limited level of customization. These suites are integrated in existing OpenCL development environments (IDEs), Xilinx SDAccel [19], and Intel OpenCL [20].

The OpenCL IDEs use HLS to improve the accessibility of the design of accelerators for FPGAs over traditional approaches based on VHDL or Verilog, which are time consuming and unfamiliar to most ML experts. In addition, these suites both provide libraries for Direct Memory Access (DMA), buffers, and communication channels, and abstract the underlying hardware, such as device drivers, PCIe link, interconnect, and accelerator placement [21]. These frameworks are similar to the Software ML Frameworks except they add the capability to customize the processor by tuning the overlay architecture on the FPGA.

Nurvitadhi et al. [22] describe a platform that supports multiple PCIe-connected FPGAs in a single server. They build a software stack on top of the Intel OPAE [23] and tightly couple operations on the CPU with operations on the FPGA. Their goal is to implement low-latency neural machine translation and do this by keeping the model in on-chip memories to avoid slower off-chip memory accesses. The ability to leverage multiple FPGAs makes this feasible. This work shows how to leverage the communication layer implemented with PCIe to target multiple FPGA overlays, but their scalability is limited by the number of boards available in one node.

These frameworks allow application developers to seamlessly deploy ML applications on FPGAs thanks to mature software and hardware development environments. However, on the one hand, the high level of abstraction through overlays minimizes the FPGA design time, and on the other hand, it reduces the user control of the generated hardware. With respect to the MS stack we define in Figure 1, these overlay frameworks allow some flexibility in the algorithms and limited flexibility in the hardware. Depending on the hooks available, a user can implement different supported layers to make their own customizations. The hardware flexibility is quite limited as an IP core is already generated. Some frameworks allow the user to modify the IP core through parameters, but this is generally limited in flexibility.

In this category of work, the focus is at the hardware level of our ML stack where the goal is to make it easier to generate cores for ML computations. These cores must then be integrated into a system that provides the full ML computing stack. Here, we present open source tools¹ that generate ML accelerators as third-party IPs to be integrated into FPGA projects.

CHaiDNN [24] is an ML library for the acceleration of deep neural networks on Xilinx UltraScale MPSoCs. The library provides a subset of ML operators to be synthesized with Vivado HLS and uses 6/8-bit integer arithmetic. Pynq DL [25] provides only a configurable IP for the convolution on Xilinx Zynq SoCs. FINN [26] is a framework for the implementation of binary neural networks that use a dataflow architecture. PipeCNN [27] is an OpenCL-based FPGA accelerator for large-scale CNNs and uses pipelined functional kernels to achieve improved throughput in inference computation. The design is scalable both in performance and hardware resources, and thus can be deployed on a variety of FPGA platforms. HLSLibs [28] is a set of libraries implemented in standard C++ for bit-accurate HLS design. Many of the library operators (e.g., MatMult, SoftMax, Sigmoid) can be easily integrated into the design of ML accelerators. Recently, CNN implementations similar to the design in this article have been produced for low bit precision CNNs [29] and for sparse CNNs [30]. These solutions provide more flexibility as the developer, in some cases, can modify the generated cores, as well as integrate additional circuitry around the provided IP cores. However, this design flow is only accessible to those with hardware design knowledge.

With respect to the ML stack, ML core generators provide full flexibility of the hardware and supported algorithms. However, they provide very little when it comes to support for integrating systems at a much larger scale, as it is the user’s responsibility to integrate the generated cores into their larger design.

In a multi-FPGA cluster, all FPGAs are network connected, and Brainwave [2, 3] built on top of Microsoft’s Catapult network-connected FPGA framework [1] is the most successful and well known. Each FPGA contains a customizable overlay. The focus of Brainwave is to minimize latency. Thus, the entire processing only uses on-chip memory and resources, and the neural network is partitioned across multiple FPGAs accordingly. The links between the network-connected FPGAs use Catapult’s 40-Gb/s custom Lightweight-Transport-Layer, a lightweight reliability layer on top of a communication protocol similar to that of UDP. When characterizing Brainwave using the stack defined in Figure 1, it can be observed that Brainwave also provides a flexible Application layer as multiple types of neural networks are supported. Brainwave is limited to the Lightweight-Transport-Layer for cluster communication between FPGAs, but this is still an improvement over frameworks that force all accelerator communication through a CPU. Finally, Brainwave provides some flexibility at synthesis time to customize precision, vector size, number of data lanes, and the size of the matrix-vector tile engine. These works allow users to scale a large ML framework across multiple nodes, providing the cluster deployment layer in the ML stack. These works also support a number of layers allowing for the user to customize their algorithm. However due to the scale, there is little hardware flexibility, as the parameterization happens at the node level as opposed to the level of the IP core.

Although AIgean can be used with a single FPGA, it best fits the category of Section 2.5, or ML Computing on Multi-FPGA Clusters, and has a similar goal as Brainwave. Both platforms use network-connected FPGAs in a peer-to-peer configuration. Brainwave uses a programmable overlay that has some parameterization that can be invoked at the time the overlay is synthesized and can implement many different ML networks depending on the program that is loaded. AIgean synthesizes custom hardware cores and implements each ML network directly in hardware, so changing an ML network will take much longer than recompiling the program for an overlay. With AIgean, there is the ability for researchers to experiment at the hardware implementation layer with hls4ml, the possibility to experiment with the communication protocols used, and to specify how the computation kernels are deployed. All of the related works provide some layer of the ML stack defined in Figure 1. AIgean is the only work that can provide support at all of these layers, allowing users to parameterize at the IP core level and at the cluster level, and support many algorithms. This is all made possible by the layered approach used by AIgean and because everything is available as open source.

We need a way to implement hardware ML-inference cores that can take specifications from common ML frameworks. We choose hls4ml [4] because it can translate the specification of ML models from common frameworks such as Keras [31], PyTorch [32], ONNX-formatted models [33], and the quantized version of KERAS, QKERAS [34], into Register-Transfer Level (RTL) implementations for FPGAs using HLS tools [35]. In our experiments, we use Vivado HLS [36] as the hls4ml back-end though the flow can be extended to other HLS tools. Hls4ml currently has support for Vivado HLS, Quartus HLS, and Mentor Catapult HLS [37].

At the start of the AIgean development, hsl4ml was a tool that was only targeted to implement ML applications that fit on a single FPGA. In this section, we describe the baseline capabilities of hls4ml, and in Section 4.2, we describe the changes we made to integrate hls4ml into AIgean. A more detailed description of the changes to hls4ml is found in Appendix A.1.

An ML designer prepares a neural network for a specific task, such as image classification, in Keras or PyTorch. After an iterative training phase that ends when the target accuracy/error goals are met, the ML designer releases a final model to be deployed for inference. The model is usually described as two files in standard formats: a JSON file for the model architecture, and an HDF5 file for the model weights and biases. These are the inputs for hls4ml. At this point, a hardware designer can fine-tune the hls4ml project and push-button translate it into a complete Vivado HLS specification (C++ and TCL files) to be synthesized and implemented for a target FPGA.

The hardware designer faces the challenge of creating an optimal FPGA implementation from the given ML model. The hls4ml framework exposes a crafted set of configuration parameters (HLS knobs) to balance the FPGA resource usage and the latency and throughput goals. The design of ML-inference accelerators using HLS is simplified with hls4ml by hiding the large variety of HLS knobs and providing carefully optimized layer implementations for HLS.

The conversion from deep learning model to HLS-based software is done by constructing a custom intermediate network representation that is amenable to low-latency design. From this intermediate representation, HLS code is generated with design guidelines specified in a configuration file. Optimized HLS implementations of neural network layers are generated, with the optimization dependent on specified configuration parameters. The code is thoroughly modular, and most optimizations can be tuned after the HLS code generation. Hls4ml has been used to construct MLP networks, CNNs, Graph neural networks, RNNs, and BDTs [38, 39, 40, 41].

The hls4ml design flow explicitly focuses on batch-1 processing. Larger batch processing is not considered. The design flow is similar to the FINN architecture [26, 42] in that model-specific layers are implemented. A critical element of the design of hls4ml is to allow for very low latency implementation of ML algorithms with a low initiation interval.² As a consequence, hls4ml generates an HLS firmware implementation of the neural network on a layer-by-layer basis. Each layer corresponds to a different firmware block, and therefore individual layers can be run concurrently. This design paradigm differs from most other FPGA deep learning implementations, such as Xilinx ML Suite, where the same firmware blocks are repeatedly used to perform the inference computations over many layers of a neural network. Separation of the layers into separate firmware blocks is particularly tractable for multiprocessor use since layers can easily be split into separate IP blocks without any modifications in the algorithm design or changes in resource usage. We leveraged this capability for AIgean.

The trade-off among latency, initiation interval, and resource usage determines the parallelization of the accelerator logic (and vice versa). In hls4ml, this trade-off is configured with a single configuration parameter—the reuse factor. The choice of a value for the reuse factor affects the initiation interval of the RTL pipelines and the number of critical resources (e.g., DSPs) in each layer of the neural network.

Within the hls4ml implementation, the reuse factor dictates the number of times a single DSP is reused within a single matrix multiplication. This factor translates directly to the number of resources that each layer uses. In particular, both the DSP usage and the number of BRAM partitions will scale with the reuse. By scaling the reuse factor value, the designer can explore various implementations. A reuse factor of 1 generates a completely parallel implementation (lowest latency); a reuse factor of R, generates an implementation with 1/R fewer DSPs (lower resource usage) and BRAM partitions. Designers may choose a larger value for reuse factor in the case of limited resources and a smaller value when they can afford higher parallelism.

For the development of AIgean, a number of improvements were made within hls4ml. These developments include:

Finally, the generated ML accelerators have interfaces that are system agnostic. In Section 4, we illustrate our extension to the Galapagos flow that enables a designer to rapidly prototype ML accelerators and deploy them in a Galapagos system with minimal effort.

AIgean is a development stack for deploying ML applications across multi-FPGA and CPU clusters. This logically can be seen as a superset of the Galapagos development stack with a specific application layer. Galapagos is a hardware stack that provides customization at different levels of abstraction [6]. The main goal of Galapagos is to abstract the low-level hardware plumbing required to deploy an application across multiple FPGAs while also providing the ability to port applications across multiple FPGA platforms (i.e., platforms built using different FPGA cards with different networking infrastructures). We know of no other platform that can take as input just the computation kernels and a logical description of the connections between the kernels and then generate all of the FPGA bitstreams with all of the network connectivity included. Without Galapagos, an application developer with a multi-FPGA cluster would need to be an expert in hardware design. In addition to building the computation kernels, the developer would need to incorporate into their design the interfaces to the on-board memory, the network interfaces, the network protocol hardware (most likely hardware UDP or TCP/IP cores), and configure Ethernet MAC addresses, IP addresses, the routing information for moving data between kernels, as well as build all of the packet formatting and protocol translation between the computation kernels. The FPGA vendor platforms for OpenCL [19, 20] are usable by non-hardware application developers because they abstract away these details. Galapagos does the equivalent abstraction, but for a multi-FPGA cluster environment. By building on Galapagos for AIgean, we can leverage the multi-FPGA abstraction that is provided by Galapagos, and can focus on the integration of hls4ml and not worry about the low-level hardware plumbing required.

The structure of Galapagos is analogous to a traditional software or networking stack, with each layer of the stack providing an API for the layer above. The lower the layer in the stack, the closer it is to the physical hardware. Figure 2 shows the AIgean stack.

Physical hardware and connectivity. This layer represents the physical hardware that runs applications, and for this work we focus on the FPGAs. Aside from implementing the computations in FPGA logic, we can also implement different forms of connectivity. In Galapagos, we can use PCIe, 10G SFP+ Ethernet, 100G QSFP28 Ethernet, and L1 circuit switching. For Ethernet, we can select TCP/IP, UDP, and raw L2 Ethernet. Once configured in this lower level, typical software and ML practitioners can work at a higher level of abstraction.

Hypervisor. The hypervisor³ abstracts away the I/O interfaces of a single FPGA so that the hardware applications only needs to connect to a standardized interface, and they can then be implemented on any FPGA that has the same hypervisor. This is the key requirement that enables applications to be portable across multiple hardware platforms enabled with Galapagos. In the same way, the hypervisor in the software world provides an abstraction of the hardware and some level of services, typically I/O and memory.

Middleware. This layer connects the different devices within the Galapagos cluster and sets the off-chip network communication protocols between them. Within Galapagos, computation kernels can address each other and are agnostic of their placement.

Communication layer. The communication layer provides the APIs with the ability to send packets using the connections laid out by the middleware. Galapagos transmits packets using the AXI-Stream protocol [43]. All of the kernels within the cluster can reach any other kernel via AXI-Stream. Since the middleware layer provides the network address translation functionalities to convert AXI-Stream into off-chip network packets using the desired network communication, the network details and locations of kernels are abstracted away from the user. In software, this is the role of network-socket libraries or other network and communication protocols such as the Message Passing Interface (MPI) [44].

Application layer. For AIgean, the application layer is the ML layer provided by hls4ml and the tools used to generate the inputs to hls4ml.

The stages of the AIgean tool flow are visually presented in Figure 3. The AIgean automated flow provides a black box that takes an ML model to a CPU/FPGA cluster. In the following sections, we highlight the inner components of the black box because they can be modified by domain experts working within each part of the tool flow to explore a large design space relevant to their interests.

Each of these stages corresponds to layers of the abstraction stack for common ML frameworks we described in Figure 1. The stages are described as follows.

The full details of the modifications implemented in hls4ml are specified in Appendix A.1. In particular, we modified hls4ml to produce HLS cores with streaming interfaces so that they can fit with the streaming model of Galapagos. As part of these modifications, an auxiliary channel is added between layers to allow for network inference to reset in the middle of an inference. This additional option is useful for multi-FPGA implementations where data streams are vulnerable and can be interrupted. Additionally, hls4ml was extended to have the option of large CNN layers with millions of weights. The previous CNN implementation was intended for low-latency use and could not support as many weights. The core of hls4ml, including the fully connected layers and activation functions, remain the same and are embedded in the streaming implementation. As a consequence, the full functionality of hls4ml is preserved in this streaming implementation, allowing for a broad range of models to be implemented.

Further optimizations are applied for ResNet-50, including the fusing of batchnorm layers with the convolutional layers and the compression of 8-bit weights into single 16-bit weights so that DSP multiplier units can be used and the total number of needed multiplications is halved. Finally, an additional configuration parameter is added to the autogeneration that allows for approximate tuning of the reuse factor to obtain the desired CNN throughput. With this new option, the tuning factor for the network is defined by the desired throughput in operational clocks and the reuse factor is adjusted so that every layer achieves the desired throughput. As a result of the throughput tuning, the reuse factor will be adjusted to ensure the inter-layer latency is roughly the same. A balanced throughput avoids significant bottlenecks between the layers. The full details of the throughput tuning is described in Appendix A.1.

The hls4ml streams have no side channels. There are only the data payload (e.g., fixed-point data) and ready/valid signals between kernels. This kind of stream suffices for point-to-point connections within one FPGA. However, off-chip communication between cores requires additional routing information. Galapagos IP cores use HLS streams with side channels to provide routing information (i.e., destination). The destination field that is used in Galapagos by default is 16 bits, but this is configurable depending on the number of IP cores we have in our cluster. We designed bridges to transform hls4ml streams to Galapagos streams by adding the additional routing information and packing the data in larger-bit-width Galapagos streams. These bridges convert a single tensor consisting of many parallel AXI streams, with one stream per dimension, into a single large bit width stream for off-chip communication. Since the bridge’s size is dependent on the hls4ml IP core (the bridges input size depends on the number of streams), this also needs to be auto-generated. A visual representation of this can be seen in Figure 4.

Furthermore, with the processing of streams, it allows us to send flits of data corresponding to different images within the same packet, allowing for a more efficient use of bandwidth. These streams are also configurable by allowing the user to configure how many AXIS stream flits⁴ to pack within one network packet. This solution can be significant for FPGA-CPU links where it is crucial to amortize the cost of network communication on the CPU, due to the overhead added by the Linux network stack.

To explore the design space of large ML networks (like ResNet-50) across multiple FPGAs, we developed an automated partitioner to work with the rest of the Galapagos framework. When we turned to ResNet-50 to implement a very large network, it quickly became clear that we needed an automated means for partitioning a large application to make the best use of the resources. Our first partitioner is described in Section 4.1.3 and is not specific for just hls4ml kernels but can be used for any streaming kernels that require placement.

The original Galapagos framework supported 10G TCP and L2 Ethernet for off-chip communication. We designed a bridge to provide the option for 100G UDP cores to increase the performance of our network links and to reduce the probability of the network communication being the bottleneck. This enhances the capability of any application using Galapagos, not just AIgean.

To support 100G, we also improved the portability within Galapagos to support multiple bit-widths of data. The prior bridging within Galapagos assumes all kernels communicate over AXI-stream with a destination side channel. However, due to the additional required bridge needed to allow hls4ml kernels to communicate over AXI-stream with a destination side channel, we needed to modify Galapagos to include support for the insertion of application-specific bridging. This is shown in Figure 5. For more details on the bridging provided within Galapagos, please refer to Appendix A.3. In this case, one of such application-layer bridges is the hls4ml bridge described in Section 4.2. An application-layer bridge transforms an application-layer protocol into Galapagos packets. There is a configuration control path for the user to adjust the properties of the bridge. For the network bridge, it allows users to adjust the routing between kernels by allowing the user to adjust the mapping of destinations to IP addresses.

A major goal of AIgean and Galapagos is the ease of development. Galapagos offers functional portability of cores between hardware and software by implementing software libraries to model the hardware routers and bridges shown in Figure 5. The software library (i.e., libGalapagos) is described in the work of Tarafdar and Chow [45]. Furthermore, Galapagos allows fast simulations of the cluster thanks to the combination of the HLS IP cores (in C++) and libGalapagos for the connections between cores. When we designed the additional cores and bridges for AIgean (e.g., 100G UDP core), we also implemented the libGalapagos equivalent of these cores to maintain functional portability. On top of prototyping the entire cluster in software, we have also added the ability to simulate the entire cluster in RTL. We designed an RTL model of a network switch that is configurable and can simulate the latency between network links. This capability has been invaluable during the development of AIgean as a platform but would not be required during the normal use of AIgean. The combined efforts of both these frameworks result in a fully configurable design space exploration tool of multi-node heterogeneous ML clusters.

Our hardware testbed comprises Supermicro servers with Intel Xeon E5-2650V4 CPUs and 64 GB of memory. The FPGA boards we have available are Alpha Data 8K5s with a Xilinx KU115-2 FPGA, Fidus Sidewinders with Xilinx ZU19EG FPGAs, and Xilinx Alveo U200 and U250 cards with XCU200 and XCU250 FPGAs, respectively. We have 16 Sidewinders mounted in a 16-slot PCIe chassis, and the other boards are mounted in PCIe slots of our servers. For the network interconnect, we have two Dell S4048-ON 10G and two Z9100-ON 100G switches. The servers are connected to a 10G switch and the FPGA boards are mostly connected to 100G switches. For the AIgean tests reported here, we used Vivado 2019.1 and Sidewinder FPGA boards connected to 100G switches.

The SDAccel platform we used was on an Amazon f1.2xlarge instance using SDAccel v2018.2. For those tests, hls4ml was used to generate the cores, and they were invoked as OpenCL kernels using SDAccel. The GPU tests used an Nvidia 1080Ti.

While working on this article, we have gone through several iterations of different layers of the stack. One iteration involved optimizing our HLS library to use DSPs more efficiently, with the same functionality. We describe the results in Section 5.5, but we would like to discuss the steps required to change our cluster implementation between the two IP core implementations. This change was done in the hls4ml level, particularly the domain of a hardware expert looking to optimize the hardware implementation of a particular IP core. Following the change, we ran hls4ml generating a directory structure and a makefile, with a subdirectory per IP core. At this point, we can build at the top-level makefile by typing “make,” and this will rebuild all IP cores that have changed their implementation. Then we point our IP core directory to the rest of the AIgean flow and type “make.” This will then partition the IP cores, add the bridges, and generate the bitstreams. This case study shows that an expert in the IP core generation only has to focus on their layer of abstraction and then rebuild the entire cluster by typing “make” twice.

In this section, we present the latency and throughput measurements for different link configurations. This is to provide some understanding of the penalties for communication over network links. For communication between nodes, we can use a 100G UDP core [46] or a 10G UDP core [47] on the FPGA. These are interchangeable within our framework by the user. The 100G core uses a 512-bit interface as compared to the 64-bit interface for the 10G core. The CPU NIC we use is a 10G SFP NIC [48], even when communicating to a 100G FPGA. The specific FPGA board we are using is the Fidus Sidewinder with an MPSoC FPGA [49]. For latency measurements, we send a single flit of data (8 bytes) using the four different types of links listed in Table 1. The results in Table 1 involving software are shown with the 100G UDP core on the FPGA, but similar results are observed when using the 10G UDP core. From hardware to software, we observe the FPGA outputting at 100 Gb/s, but we experience packet drop in the software when doing the throughput measurement. Observe that the links involving software are limited by the CPU network stack and library implementation, whereas the FPGA-to-FPGA links can transfer at the full network bandwidth. Note that the results in Table 1 are for the raw throughput, including the protocol headers, which is why it is possible to achieve the full link bandwidth when using the FPGAs.

Here we describe our first small multi-FPGA network implemented with AIgean. We consider an example network with applications for high-energy physics. Specifically, our network is an autoencoder designed to detect anomalous events, potentially from new physics. An autoencoder is an unsupervised learning technique that leverages a neural network where a bottleneck in the shape of the network forces a compressed representation of the original input. Details about the model and use cases can be found in Appendix A.2.

This network is a very interesting size for our studies, as it can be implemented on a single FPGA, but this requires a high degree of resource reuse that necessarily increases the inference latency. When splitting the network across multiple FPGAs, we can adjust the throughput and latency of the network by changing the reuse factor and compiling the network across multiple FPGAs. The network split across multiple FPGAs will have a higher throughput but incurs some latency from the transfer of the intermediate results.

The resources for the autoencoder network are shown in Table 2 along with the resources available on the FPGAs we used. To test this autoencoder, we considered two separate implementations of the network: an implementation using an AWS F1-instance (VU9P FPGA) using SDAccel, and a second implementation using AIgean on three Sidewinder (ZU19EG FPGA) boards. What is notable is that the single FPGA implementation would not be able to fit on a single Sidewinder board, and it would have to be spread over multiple FPGAs for the chosen reuse factor. The single FPGA implementation also requires more than one super logic region, and as a consequence has difficulty meeting timing when compiled on the F1 instance with SDAccel.

Table 3 highlights the results from implementing the autoencoder on various devices as well as on a single FPGA using SDAccel and three FPGAs using AIgean.

Our 1-FPGA autoencoder is clocked at 125 MHz at a low reuse factor when using SDAccel. Limitations in our version of SDAccel, as well as the resources required for the FPGA, prevented us from using a higher clock speed. For the 3-FPGA version, we used AIgean and were able to achieve 200 MHz for two of the FPGAs and 190 MHz for the third one. We did not try to improve it, so we will use 190 MHz since that is the limitation. To make a fair comparison to the 1-FPGA implementation, we scale the AIgean latency by the ratio of clock speeds and get \(0.08 \times 190/125 = 0.12\)ms, which is still \(0.24/0.12 = 2\) times better than the latency using SDAccel. This shows that there is still a significant architectural advantage to using multiple FPGAs and is not unexpected because more resources are available. The performance increase with three FPGAs can be attributed to (a) the use of networking to directly communicate with the FPGA, yielding low latency, and (b) less demanding resources per FPGA since only one-third of the model is implemented on each FPGA.

The implementations of this model on both a single FPGA and the full three FPGAs have an initiation interval of 552 clocks and require roughly the same resources (the reuse factor is the same). In other words, the three FPGAs are capable of processing a new image every 2.76 \(\mu s\) (362 KHz). Such a throughput approaches the demands needed for real-time processing of anomalies at the LHC. Although the single FPGA implementation with SDAccel has a potential throughput that is half that of the 3-FPGA implementation, achieving this throughput would require efficiently buffering the inputs and outputs by sending larger batches of calls on and off the FPGA through the DDR and PCIe transfers. As a consequence, the individual (batch-1) latency would be significantly degraded for the final throughput to approach half that of the 3-FPGA implementation.

To test AIgean on a much larger network, we have developed a multi-FPGA implementation of ResNet-50 [50]. The flexibility provided by AIgean allows us to target a high throughput implementation whereby we unroll the multiplication in each CNN layer at a rate corresponding to the number of pixels that are being used in each respective CNN layer. This allows for the design of ResNet-50 that can be balanced across the different CNN layers to have a uniform throughput.

Most of ResNet-50’s architecture can be broken down into many sub-blocks consisting of a Split, two to three convolutions followed by a Relu, and an addition operator as shown in Figure 6. The dashed boxes represent the IP block granularity that we have used within our implementations.

We have two implementations of ResNet-50: the first requires 12 Sidewinder boards using int-8 precision (ranging from 80% to about 90% of the resources used on each FPGA); the second is more DSP efficient and requires 10 Sidewinder boards using int-8 precision as well. We have one FPGA available to use as a 100G data generator that can feed inputs at line rate to the FPGAs. For the 12-FPGA configuration, we tested in a piece-wise fashion.⁵ We have tested the traffic generator and the first 10 of 12 FPGAs followed by testing the traffic generator and the remaining 2 FPGAs. We have verified that the full 10-FPGA configuration and the piece-wise 12-FPGA configuration can run at 660 images per second.

Table 4 summarizes the throughput and latency results of our full 12-FPGA implementation of ResNet-50. When the source data is coming from the CPU, we observe that the maximum throughput is only 400 images per second with a latency of 7 ms due to the bandwidth limitation between the CPU and the FPGA (5-ms latency between the CPU and the FPGA). To demonstrate the full performance achievable with the FPGAs, we use the FPGA data generator and observe a throughput of 660 images per second with a latency of about 1.9 ms. The latency is determined through a simulation of the full ResNet-50 network where each layer is separately run in parallel. The network delay between each FPGA is estimated from Table 1 using the QSFP. For 10 hops, the total network delay would be 0.0017 ms, which is insignificant compared to the computation latency. The next row gives the values for Microsoft’s Brainwave [51]. For the latency of Brainwave, we quote the end-to-end latency determined from sending an image to a Brainwave server and then receiving the result for a CPU within the same computing cluster. The final row shows the performance for an Nvidia V100 GPU using the mixed precision implementation of ResNet-50 applied for batch 1. The latency and throughput quoted is obtained through the use of the Triton inference server with a client on the same machine. As a consequence, the latency numbers include the PCIe transfer time in addition to the network inference. Equivalent numbers quoted by Nvidia yield a batch-2 latency of 1 ms with a throughput of 2,000 images per second for the same model [52]; batch 1 latency is not quoted.

Table 5 summarizes the resources used for our 12-FPGA implementation. Note that this was partitioned with our greedy partitioning scheme that uses a heuristic of 80% utilization before allocating the next FPGA. The 10-FPGA configuration is very similar in terms of resources but with half the DSPs. Some other noteworthy details are that a number of the layers early in the network are smaller, and we can see that the FPGAs are DSP limited as compared to the larger layers later in the network being logic limited. The highest resource utilized for each FPGA is shown in bold, representing the limiting factor of each FPGA (with exception of the last FPGA that is not fully used.). For perspective, the total resources available on an individual FPGA are shown at the bottom of Table 5. This FPGA is approximately equivalent to a single SLR of the VU9P FPGA in the Amazon F1 instance (each VU9P having three SLRs) [53]. For further perspective, we can also compare this to the Xilinx Alveo U250 [54]. Our current utilization is DSP limited, and we could fit our entire ResNet-50 implementation on two Alveo U250 boards, where the U250 board has 12.2K DSPs.

Last, we would like to contrast this implementation with previous implementations of ResNet-50. The design flow of AIgean differs from previous 8-bit implementations of ResNet-50 in that no overlay is used, and each layer is implemented separately. In this scenario, it is possible to continuously stream images through the implementation without having to wait for an image to be complete. With the overlay architecture, the images are streamed through each layer to a buffer and then subsequent layers are loaded and the next layer is streamed. As a consequence, a scheme is needed for buffering of each input. Additionally, some time is needed to switch between layers. With the AIgean design flow, the whole network exists on the FPGA fabric, and so images can be continuously pumped through. This leads to a more efficient use of multiplier resources, at a cost of additional resources to route individual layers together. Since images are continuously pumped through, we achieve batch-1 streaming. Additionally, since we are continuously pumping images through, the amount of buffering between the layers is limited to just the partial image that is needed for matrix multiplications of the CNN applied to nearby pixels.

To understand the efficient use of resources, we compute the total number of multiplication operations needed for a perfectly efficient FPGA clocked at 200 MHz. With our implementation of ResNet-50, we find a total of 4B multiplications, which if we divide by 3 \(\times 10^{5}\) clocks to achieve a 1.5-ms latency at 200 MHz yields a total of 13,500 multiplications per cycle. Our current implementation uses 15,419 DSPs, which is slightly more due to the fact that many of the individual layers are tuned to a latency that is actually below 1.5 ms. The number of DSPs can be reduced through two means: first, through the sharing of DSPs, which is only partially implemented here, and second, through the use of a faster clock frequency. The sharing of DSPs would lead to roughly a factor of 2 reduction in DSPs. A faster clock frequency would yield a lower latency for the same number of DSPs. Since each multiplier unit is mapped directly to a specific multiplication within the network, the only way to inefficiently use the DSP resources results from the case where an allowed reuse parameter for a specific latency is not near the desired throughput and, as a consequence, the individual layer has a significantly lower latency than its neighboring layers.

Adjustment of the reuse parameter effectively modifies the initiation interval of each layer. A reuse factor of 5,000, corresponds to a layer that has an initiation interval of 5,000. To efficiently adjust the reuse parameters with hls4ml, the reuse needs to split the dense matrix multiply embedded within the layer across DSPs so as to maintain a regular systolic-array architecture. As a consequence, optimal implementations of the reuse can only be certain numbers, which is determined by the number of input and output features of each layer. Our current implementation is near ideal since 1.5 ms allows for a consistent set of reuse values that are near the 1.5-ms ideal latency point. To achieve a higher throughput, we need to adjust the reuse factor to the desired throughput and re-implement the whole design. Although this procedure requires a lot of computing, the whole procedure is automated through the AIgean design flow.

When adjusting the reuse factor, we observe a direct correlation with the number of DSPs. Halving the reuse factor will halve the initiation interval of the matrix multiply within a layer, and it will also double the number of DSPs. Flip-Flops, and LUTs will not change as significantly since they largely exist to store partial images. BlockRAMs are used primarily to store weights of the neural network on the FPGA. Their second use is to act as a buffer between layers. As a consequence, the BlockRAM resources will not change significantly with reuse factor. In this current implementation, since DSP sharing of the multiplications is only partially used, the resulting resources are more consistent with a ResNet-50 implementation having a latency of roughly half the observed latency (0.75 ms).

Faster implementations of ResNet-50 are possible by adjusting the reuse factor. However, for CNNs, a lower bound is present in the current, pixel-by-pixel implementation of the algorithm. The lower bound results from the fact that for each pixel that streams through the algorithm, there is a one clock latency. Furthermore, there is an additional latency of three clocks to prepare the inputs to run the matrix multiplication. For layers within the network, where there are many pixels, such as the first layer, the ultimate latency is limited by these operations. Applying this limit to the first layer of ResNet-50, we find that the single layer throughput is bounded to be greater than roughly 0.4 ms. Lower single-inference latencies can still be achieved by splitting the image into sub-images and simultaneously streaming these sub-image streams into separate, cloned implementations of the chosen layer. Although the use of multiple streams effectively reduces the single inference throughput by the (number of streams)\(^{-1}\), it has the added cost of increasing the resources by the number of streams.

Hls4ml was initially designed to address the necessity for ultra low latency inference. In this context, it was developed to allow for the possibility of deep neural network inference at timescales below \(1 \mu s\) that are pipelined with initiation intervals that are a few tens of nanoseconds. Such low-latency, high throughput networks are required to process information at the Large Hadron Collider in an all FPGA system with an approximate throughput of 1 Petabit/s. In this context, algorithms were explicitly designed to achieve the fastest possible latency at the cost of utilizing more hardware. In place of reusing network layers, deep neural networks are unrolled on the processor to allow for sequential, batch-1 processing of network inferences with initiation intervals smaller than the total network latency. As a result of this design flow, the focus on hls4ml was towards the development of small networks that used a large amount of resources but that could also run at very low latencies.

Extending this paradigm to larger networks, such as ResNet-50, was not considered part of the scope of hls4ml since such large networks would require resources greater than a single FPGA. As a consequence, hls4ml did not contain large layer implementations. However, with AIgean, through the distribution of a single network across many FPGAs, the possibility of extensive, low-latency networks under the hls4ml design flow is achievable. Consequently, hls4ml was extensively adapted to handle these large networks to create AIgean. In particular, we created or heavily adapted implementations of many new layers of hls4ml, including:

These layers make up the core of most deep neural networks used. Furthermore, the large layer design flow established through the development of AIgean will enable the fast implementation of other layers following the AIgean implementations. In this appendix, we will outline the various adaptations needed in hls4ml to run a wide variety of algorithms at large scales. Furthermore, we will comment on new features added to hls4ml that enhance the core software framework and enables large model, high throughput implementations.

In this appendix, we present a detailed description of each NN model used in this paper. The choice of these architectures is partly motivated by use in high energy physics, where low-latency deep neural network inference is an essential tool for operation. As a consequence, we comment on the model architecture and its application to problems within physics.

Galapagos is a heterogeneous deployment stack that allows users to deploy streaming IP cores on a cluster of FPGAs and CPUs. First we will describe the high-level abstraction model of Galapagos and then delve into each layer of abstraction that we implemented.