HitM: high-throughput ReRAM-based PIM for multi-modal neural networks

With the rapid progress of artificial intelligence (AI) algorithms, multi-modal deep neural networks (DNNs) have been applied to some challenging tasks, e.g., image and video description to process multi-modal information from vision and language. Resistive-memory-based processing-in-memory (ReRAM-based PIM) has been extensively studied to accelerate either convolutional neural network (CNN) or recurrent neural network (RNN). According to the requirements of their core layers, i.e. convolutional layers and linear layers, the existing ReRAM-based PIMs adopt different optimization schemes for them. Directly deploying multi-modal DNNs on the existing ReRAM-based PIMs, however, is inefficient because multi-modal DNNs have combined CNN and RNN where the primary layers differ depending on the specific tasks. Therefore, a high-efficiency ReRAM-based PIM design for multi-modal DNNs necessitates an adaptive optimization to the given network. In this work, we propose HitM, a high-throughput ReRAM-based PIM for multi-modal DNNs with a two-stage workflow, which consists of a static analysis and an adaptive optimization. The static analysis generates the layer-wise resource and computation information with the input multi-modal DNN description and the adaptive optimization produces a high-throughput ReRAM-based PIM design through the dynamic algorithm based on hardware resources and the information from the static analysis. We evaluated HitM using several popular multi-modal DNNs with different parameters and structures and compared it with a naïve ReRAM-based PIM design and an optimal-throughput ReRAM-based PIM design that assumes no hardware resource limitations. The experimental results show that HitM averagely achieves 78.01% of the optimal throughput while consumes 64.52% of the total hardware resources.