Surrogate Modeling for HPC Application Iteration Times Forecasting with Network Features

Machine learning (ML) for time-series forecasting has been studied for a long time [9]. The task aims to forecast a period of future data given a sequence of historical data and has a variety of applications, including transportation, finance and medicine. The earlier researchers leverage the statistical methods, e.g., ARIMA (AutoregRessive Integrated Moving Average) [3, 18], and traditional machine learning methods, such as SVM (Support Vector Machine) [5, 14] to forecast time-series data. However, they may not achieve desirable performance due to their linear assumptions. Recently, time-series forecasting has made a significant progress due to the emergence of the deep learning. Deep learning models deliver expressive performance because they can capture complex pattern in time-series data, like Convolutional Neural Network (CNN) [29, 34, 36], Recurrent Neural Network (RNN) [7, 22, 29], LSTM [1, 21, 34], Graph neural network (GNN) [6, 30, 37], Transformer [20, 25, 28] and State Space Models [2, 10, 32]. However, none of the above work aim at forecasting application iteration times in the HPC system. This paper is focused on forecasting application iteration times in the dragonfly system.

Various multi-resolution and hybrid PDES models [11, 12, 17, 19] have been proposed to accelerate high-fidelity PDES simulations and have shown promising results. With the emergence of machine learning, the community is interested in designing [8, 31, 33] a high-fidelity ML-based surrogate model to accelerate PDES by forecasting port-level network traffic in the dragonfly system. Different from the existing work focusing on port-level network traffic forecasting, we forecast application iteration times, a workload-level characteristics, and take network features into account to facilitate forecasting accuracy with machine learning surrogate models.

We focus on the 1D dragonfly network as shown in Figure 1. The dragonfly network [16] has a hierarchical architecture with three levels: router, group, and system. The system are divided into multiple groups, and a group has multiple routers. A router connects to computer nodes (terminal) by terminal channel links, connects to other routers within a group by local channel links and connects to other routers outside the group by global channel links. The connection points between routers and links are ports on routers. When running on the dragonfly network, applications are divided into multiple processes, which are then placed on the computer nodes. Each process occupies a computer nodes and the processes collaborate with each other on executing applications. The execution commonly consists of multiple steps, so each process has multiple iterations to complete. Each process is assigned a unique identifier known as a "rank"¹. We aim to forecast application iteration times for each rank, which refers to the time it takes to complete one iteration. Iteration denotes the repetitive execution of a set of computational and communication tasks. The processes communicate with each other by sending messages, and routers forward the messages through the ports. During the communication, the ports have key characteristics in the dragonfly network, e.g., bandwidth-consumed, busy time, etc. The system characteristics reflect the status of the dragonfly network and can be potentially used to improve the accuracy of forecasts.

Let the dragonfly network have n_r routers and n_c computer nodes and n_p processes placed on the computer nodes for execution. In the execution, the n_p processes work together to complete T iterations. For a rank p ∈ {1, 2,..., n_p} at an iteration t ∈ {1, 2,..., T}, we use y_{p, t} to denote its application iteration time, and use x_{p, t} to denote network features of a router connecting to a compute node where the rank p exists. For ease of expression, we ignore the subscript p in the following descriptions and formally define a problem:

Problem Statement.Given a application iteration times and network features sequence for a rank p with look-back window $\mathcal {B}=\lbrace y_{t-(L_x-1)},y_{t-(L_x-2)},...,y_{t}; x_{t-(L_x-1)},x_{t-(L_x-2)},...,x_{t}\rbrace$ with length L_x, we aim to forecast a sequence of future application iteration times $\mathcal {F}=\lbrace y_{t+1},y_{t+2},...,y_{t+L_y}\rbrace$ with length L_y.

We investigate four surrogate modeling approaches to forecast iteration times in a distributed setting where each compute node holding a rank has a surrogate model. The details are as follows:

LAST is a heuristic method and forecasts the application iteration time of a future iteration as the latest historical application iteration time, i.e., y_{t + 1} = y_t. Note that the length of the foretasted sequence and the look-back window both have to be 1, i.e., L_x = L_y = 1.

Average is also a heuristic method and forecasts application iteration time of a future iteration as the average of the look-back window $\mathcal {B}$, i.e., $y_{t+1}=\frac{\sum _{i=t-(L_x-1)}^{t}y_i}{L_x}$. Note that we have settled for the length of the foretasted sequence to be 1 and a length of the look-back window of 10, i.e. L_x = 10 and L_y = 1.

ARIMA (AutoRegressive Integrated Moving Average)  [3] is a classical statistical time series analysis method and can handle non-stationary time series. The look-back window L_x is 250 due to the frequent occurrence of matrix decomposition errors for small L_x, i.e. L_x = 250. The forecasted sequence has size of 1, i.e. L_y = 1.

LSTM (Long Short-Term Memory) [13] is a well-known machine learning model and widely used in time series data. LSTM is effective in addressing the gradient vanishing issue in sequential modeling problems. We set L_x = 10 and L_y = 1.

LSTM-Feat is a variant of LSTM. Different from the above surrogate modeling methods where the input and output are both application iteration times, it leverages a combination of application iteration times sequences and network features sequences to forecast iteration times sequences. The motivation is to leverage the potential correlation among network features and application iteration times. We set L_x = 10 and L_y = 1.

Network Topology. The dragonfly network (see Figure 1) has a hierarchical design, consisting of the all-to-all inter-group connection and intra-group connection. Our network has 72 compute nodes and 36 routers equally divided across 9 groups. Each router has 7 ports: 2 terminal ports, 3 local ports, and 2 global ports.

Network Simulator. We utilize CODES [23] to simulate our workload. Times are collected for 2000 iterations along with network features from each port collected at 250 μ s.

Network Features. The network features consist of bw-consumed, qos-data, busy-time, vc-occupancy, and downstream-credits. Qos-data is the amount of data sent by the port; bw-consumed is the percentage of the consumed bandwidth; busy-time is the total time the port was stalled, i.e. chunks were blocked from sending due to flow control; vc-occupancy is the number of bytes in each VoQ buffer of the port at the point in time when the measurement was taken; downstream-credits is the number of credits available for the respective downstream virtual channels at the point in time when the measurement was taken.

Align Datasets. We align application iteration times and network features datasets to resolve the inconsistency between them. Iteration times are on compute nodes while network features are on routers; an iteration time is recorded when an iteration completes while network features are collected at a fixed time interval. To eliminate the inconsistency, we combine the iteration time on a compute node with network features on a router connecting to the compute node; we search time points in network features dataset that are the nearest to iteration times, and take the network features of the time points to combine the iteration times.

HPC Workload. The workload includes: (1) MILC is a HPC application used to study quantum chromodynamics (QCD) and features numerous nonblocking send/receive communication operations. (2) UR is a synthetic traffic featuring each node sending successive messages to a random destination. The messages are streamed at user-defined injection loads, alternating between 10% and 100%.

Job Placement. We investigate two job placements: (1) Contiguous Placement selects computer nodes consecutively for the processes of the job to occupy. (2) Random Placement selects computer nodes randomly for the processes of the job to occupy.

Routing Policy.Progressive adaptive routing [27] is used in our simulation. Packets are routed along minimal or non-minimal paths based on the network congestion state. When a non-minimal path is selected, the packet will be minimally routed into a randomly intermediate router, and then minimally forwarded to its destination. According to job placement and routing policy strategies, we denote the two settings as cont-adp and rand-adp, respectively.

Evaluation Metrics. We use metrics MSE (Mean Square Error) and MAE (Mean Absolute Error) to assess the effectiveness of models [34, 35], and I.T.O. (Inference Time Overhead) measures time overhead when the surrogate models do forecasts.

Implementation Details. We split 2000 iterations into training, validation, and test data. The split ratio is 6:2:2 for cont-adp and 7:1.5:1.5 for rand-adp due to their different distributions. We normalize iteration times into range [0, 1] for stability of training. We run experiments 3 times and report the average and standard deviation.

Application Iteration Times Forecasting. We show the performance comparison for the foreasts in Table 1. According to the experimental results, we have the following observations:

• In terms of MSE and MAE, the LSTM-Feat outperforms other surrogate models, including heuristical methods, i.e., LAST and Average, traditional statistical method, i.e., ARIMA, and deep learning methods without considering network features, i.e., LSTM. For instance, LSTM-Feat outperforms LAST and LSTM by $50\%$ and $35\%$, respectively, w.r.t. MSE. It demonstrates LSTM-Feat can capture temporal patterns in iteration times sequences and dependencies between network features and iteration times.
  
• In terms of I.T.O., LAST and Average are the two fastest methods as the implementation of the heuristic methods are simple. ARIMA is the slowest method as such a statistical method needs to calculate parameters by fitting a set of data in a long look-back window per step. The efficiency of deep learning methods are between the above two kind of methods. For exmaple, LSTM-Feat is slower than LAST but faster than ARIMA. However, we show deep learning is potential w.r.t. I.T.O. in the sensitivity analysis.
  

Feature Importance Analysis. We conduct feature importance analysis in Figure 2. In detail, we remove a specific network feature across all ports of a router, e.g., bw-consumed or busy-time, and remove all features from a type of ports of a router, e.g.., local or global ports, to observe the performance change of the LSTM-Feat. Accordingly, we observe the following points:

• Network features are generally helpful to iteration times forecasting. For example, in the rand-adp configuration, if we remove bw-consumed feature, MAE increases from 0.0167 to 0.0191. It shows dependencies exist between bw-consumed and iteration times. On the other hand, we also note that vc-occupancy and downstream-credits are useful in cont-adp configuration while they cannot help forecast iteration times in rand-adp configuration. It may be because both features capture the state of port buffers for a single point in time during an iteration unlike other features that capture the behavior of the port throughout the iteration. Furthermore, adaptive routing causes buffer occupancies to vary somewhat stochastically, making it difficult to associate occupancy trends to specific per-rank application activities.
  
• Network features on all types of ports of a router are important to forecast application iteration times. For instance, in the cont-adp configuration, MAE of LSTM-Feat increases from 0.0253 to 0.0261 with the removal of network features on local ports.
  

Sensitivity Analysis. We investigate the impact of the length L_y of the future values sequence $\mathcal {F}$ on LSTM-Feat. Particularly, we vary the L_y and record values of metrics as shown in Figure 3. We observe there is a trade-off between effectiveness (MSE) and efficiency (I.T.O.) for LSTM-Feat. When the length L_y increases, MSE generally increases but I.T.O. consistently decreases. It means the effectiveness of LSTM-Feat decreases but the efficiency increases with the L_y increasing. For effectiveness diminishing, the reason is that forecasting farther length at a time is more difficult due to more uncertainty in the father length. With regard to efficiency increasing, it is because the required inference times are reduced if increasing forecasting length at a time given the fixed total forecasting length. For instance, if the total forecasting length is 200 and the L_y is 1, the required inference times are 200; if the L_y is 10, the required inference times are reduced to 20. It shed light to a promising direction where deep learning methods can achieve both satisfactory effectiveness and efficiency if choosing appropriate L_y.

Visualization. The visualization of a rank is shown in Figure 4. LSTM-Feat can achieve satisfactory forecasting performance.