An anomaly detection model for multivariate time series with anomaly perception

Classic algorithms

Because of their simplicity and interpretability, classical machine learning algorithms have been favored by some studies in the past for anomaly detection tasks. For instance, Ramaswamy, Rastogi & Shim (2000) introduced a new method for identifying outliers using the distance from a point to its k-th nearest neighbor. Points were ranked based on this distance, and the top n points were classified as outliers. Shyu et al. (2003) employed the principal component analysis method to extract key features from the data, resulting in low-dimensional projections. The reconstruction error resulting from this process was then utilized as the anomaly score. Furthermore, Liu, Ting & Zhou (2008) proposed a novel model-based approach to anomaly detection that shifts the focus from profiling normal instances to explicitly isolating anomalies. The innovative iForest algorithm efficiently leverages sub-sampling, resulting in linear time complexity and minimal memory requirements. While these techniques offer ease of use and minimal processing complexity, their detection accuracy is somewhat limited due to the neglect of the intrinsic temporal properties inherent in time series data.

Deep learning algorithms

In recent years, with the expansion of data size and complexity, classical machine learning algorithms have faced performance bottlenecks when dealing with large-scale and high-dimensional data. Deep learning is widely used in the field of multivariate time series data anomaly detection due to its excellent feature extraction capability and superiority in modeling high-dimensional sequence data. Some recurrent neural networks, such as RNN and LSTM, are widely used in time series data anomaly detection due to their natural sequence feature extraction capability. For example, Su et al. (2019) proposed a stochastic recurrent neural network that captures the normal patterns of multivariate time series by modeling the data distribution using stochastic latent variables. Wei et al. (2023) proposed a hybrid model based on LSTMs and self-encoders that solves the long-term dependency problem, which cannot be solved by shallow machine learning. Due to its efficient parallel computing capabilities and powerful context information processing abilities, Tuli, Casale & Jennings (2022) chose to build an anomaly detection model based on Transformer. This model can leverage the characteristics of Transformer to quickly and accurately identify abnormal observations in time series data. In addition, in order to explicitly capture the dependencies between variables, Deng & Hooi (2021) has incorporated a graph structure learning module to increase the spatial feature extraction capability of the model. To increase model performance, several studies (Zhao et al., 2020; Xia et al., 2023) have, of course, merged the first two types of approaches to extract temporal as well as spatial correlations of sequences. More recently, diffusion models (Lin et al., 2023) have attracted significant attention in the realm of artificial intelligence content generation, with some researchers applying them to anomaly detection in multivariate time series data. A noteworthy example is ImDiffusion (Chen et al., 2023), a novel framework rooted in the imputation diffusion model. It accurately captures the inherent dependencies and stochastic nature of MTS data, enabling precise and robust anomaly detection. However, a notable limitation of these techniques is that they do not fully leverage the limited available aberrant data, which could potentially further enhance the performance of multivariate time series anomaly detection.

Overview

The goal of the EDD model is to encode normal data into similar probability distributions within the latent space, while ensuring that the probability distribution of abnormal data stands out distinctly. This approach allows for the precise identification of abnormalities. The EDD model is comprised of three key components: the Encoder, the Decoder, and the Discriminator. Figure 1 illustrates the model’s fundamental structure. The Encoder is responsible for meticulously extracting the features from the multivariate time series data and encoding the normal data into a designated distribution. To ensure that the Encoder captures the essence of the original data and its output distribution incorporates the primary information, the Decoder is employed. The Decoder reconstructs the original data using samples drawn from the Encoder’s probability distribution. The Discriminator of EDD is proposed to discern the disparities between the distributions of normal and anomalous data in the latent space. Its guidance also assists the Encoder in determining which type of data (specifically, anomalous data) does not require encoding into the designated distribution. Each of the three modules is comprehensively explained in the following subsections.

Encoder

The Encoder is responsible for extracting relevant features from time series data. In analogy to VAE (Kingma & Welling, 2013), the EDD model aims to map these extracted features onto a Gaussian distribution. Over the long term, the dependencies among multivariate time series data remain consistent, with normal data adhering to these dependencies while abnormal data deviate from them. Consequently, theoretically, anomaly detection can be achieved by isolating the characteristics that are shared by normal data but absent in abnormal data. When extracting features from multivariate time series data, it is essential to capture both temporal and spatial features. Temporal features refer to the relationships between individual variables across different time points, elucidating the evolving trends within each sequence. Spatial features, on the other hand, highlight the interconnections among various variables. Accurate extraction of these two types of features enhances the model’s generalization capabilities, enabling a more precise distinction between normal and abnormal data.

Prior research (Dos Santos & Gatti, 2014) has demonstrated the effectiveness of convolution operations in feature extraction. Therefore, we initiate the feature extraction process by applying a 1-D convolution layer to the original data. This step aids in preprocessing the data before proceeding to extract the aforementioned temporal and spatial features.

Decoder

In our EDD model, a subset of data is randomly sampled from the Gaussian distribution formulated by the Encoder, leveraging the reparameterization strategy. These sampled data are subsequently fed into the Decoder, which serves as a multi-layer perceptron responsible for reconstructing the original data. The Decoder plays a pivotal role in guiding the Encoder towards extracting richer, more meaningful features. While the Decoder is not directly involved in anomaly detection, its significance lies in providing a mechanism that enables the Encoder to delve deeper into the inherent differences between normal and abnormal data in the training dataset. Without the Decoder, the Encoder may overrely on superficial features, disregarding the deeper, more significant features within the data. This limitation may compromise the model’s generalization capabilities. Consequently, when confronted with unseen anomalies, the Discriminator may struggle to effectively identify them if it lacks adequate extraction of internal features.

We utilize a multi-layer perceptron (MLP) as the architecture for our Decoder. By introducing randomly sampled data into this MLP, it generates a reconstructed approximation of the original data. Subsequently, we employ gradient descent to minimize the reconstruction loss, which is computed as the mean square error (MSE) between the reconstructed and the actual data. The reconstruction loss is formally defined as follows: (10)L o s s r e c = ∑ t = 1 k ∑ i = 1 N x i , t − x ˆ i , t 2 N × k .

In this formula, x_i,t denotes the actual value of the ith variable at time t within the considered time window, while x ˆ i , t represents the reconstructed approximation of the same variable at the same time point.

Discriminator

Just like the Decoder, the Discriminator obtains its input via reparameterization, which involves randomly sampling data from the Gaussian distribution generated by the Encoder. The core responsibility of the Discriminator is to accurately differentiate between normal and abnormal data by learning the distinct probability distributions of both types in the latent space. Additionally, since the gradients of the sampled data are not truncated, the Discriminator effectively guides the Encoder to refrain from encoding abnormal data into regions of high probability density. Unlike supervised learning methods that solely rely on a loss function to separate the Encoder’s output for abnormal data, our approach offers the advantage of allowing the Encoder to focus solely on learning the features of normal data. This enhanced focus on normal data features boosts the model’s expressive power, facilitating the clustering of normal data in the latent space. Consequently, the Discriminator can effectively discern the distributional disparities between normal and abnormal data in the latent space, even with limited exposure to abnormal data samples.

The Discriminator’s architecture also employs a multi-layer perceptron (MLP) for implementation. The model’s output comprises a set of vectors. To ensure precise anomaly detection, we aim for the model to produce vectors that are nearly all-zeros for normal data and close to all-ones for abnormal data. To accomplish this, we leverage the cross-entropy loss function. (11)L o s s p o s = − 1 N ∑ i = 1 N log 1 − p i (12)L o s s n e g = − 1 N ∑ i = 1 N log p i .

We employ binary cross-entropy to compute the Discriminator’s loss. where Loss_pos denotes the discrimination loss associated with normal data, while Loss_neg represents the corresponding loss for abnormal data. The model’s output is represented by p_i.

Model training

Algorithm 1 outlines the training procedure for our model. Initially, the Encoder processes the normal data and a randomly selected subset of abnormal data at time t to derive their latent space representations, denoted as Z_pos and Z_neg (line 5). Subsequently, based on these distributions, resampling techniques are employed to generate corresponding sample data (Line 6). These sampled data are then input into the Discriminator, and their respective discrimination results, O p o s d s and O n e g d s, are obtained (Line 7). Concurrently, the sampled data from the latent space distribution of normal data is fed into the Decoder to reconstruct the original normal data, yielding the reconstruction output O p o s d c (Line 8). To evaluate the model’s performance, we utilize Eq. (9) to compute the KL divergence (Loss_KL) between the normal distribution output by the Decoder and the predefined distribution, measuring their similarity (Line 9). Additionally, we employ Eq. (10) to calculate the reconstruction error (Loss_rec) of the normal data, assessing the accuracy of data reconstruction (Line 10). In terms of classification performance, we utilize Eq. (11) and Eq. (12) to calculate the binary cross-entropy losses (Loss_pos and Loss_neg) for normal and anomalous data, respectively. These loss values reflect the model’s performance in anomaly detection (Line 11). Finally, we aggregate these loss terms and update the model parameters iteratively using gradient descent algorithms (Lines 12, 13).

 
__________________________________________________________________________________ 
 
 Algorithm 1: Training Process 
    Require: Encoder E, Decoder DC and Discriminator DS, normal 
                  dataset Xpos and anomaly dataset Xneg, iteration limit n. 
  1  Initialize weights E, DC, DS 
 2  i ← 0 
  3  while i < n do 
     4   for t=1 to T do 
    5   Zpos,Zneg ← E(Xtpos),E(Xsneg) 
6   Spos,Sneg ← Sample(Zpos),Sample(Zneg) 
7   Odspos,Odsneg ← DS(Spos),DS(Sneg) 
8   Odcpos ← DC(Spos) 
9   LossKL = KL(Zpos,Z0) 
10   Lossrec = ∥ ∥ 
 Odcpos − Xtpos∥ ∥ 
11   Losspos,Lossneg = BCELoss(Odspos,⃗ 0),BCELoss(Odsneg,⃗ 1) 
12   Loss = LossKL + LossRec + Losspos + Lossneg 
13   Update weights of E, DC, DS using Loss 
14   i ← i + 1 
15   end 
16  end 
__________________________________________________________________________________    

Model inference

Using the trained anomaly detection model for inference (summarized in Algorithm 2 ), we define the anomaly score for a given data point within a specific time window of the test set as follows: (13)S c o r e = B C E L o s s O d s , 0 →

As previously mentioned, the Decoder serves solely as an auxiliary component, aiding in the extraction of more comprehensive features from the Encoder. Consequently, during the inference phase, there is no need to feed the test data into the Decoder for reconstruction. Instead, our focus is on utilizing the Encoder to map the test data into the latent space effectively (line 2). Following this, we sample data from the latent space distribution and forward it to the Discriminator (line 3). The Discriminator then produces a set of vectors, where each value indicates the likelihood of the corresponding test data being anomalous (line 4). To derive the final anomaly score, we compute the cross-entropy loss between this vector and 0 (line 5). In alignment with established practices in prior research, we adopt the Peak Over Threshold (POT) method Siffer et al. (2017) for automatic threshold selection. The POT method is a statistical approach rooted in extreme value theory, which fits the data distribution to a generalized Pareto distribution. This enables us to determine an appropriate risk value that dynamically sets the threshold. Any anomaly score surpassing the POT threshold is deemed anomalous (line 6).

 
__________________________________________________________________________________________ 
 
  Algorithm 2: Inference Process 
    Require: Trained Encoder E, Decoder DC and Discriminator DS, test 
                  dataset X. 
  1  for t=1 to T do 
     2   Z ← E(Xt) 
3   S ← Sample(Z) 
4   Ods ← DS(S) 
5   Score = BCELoss(Odspos,⃗ 0) 
6   yt = 1(Scoret ≥ POT) 
  7  end 
 
  __________________________________________________________________________________    

Datasets and metrics

• Datasets: To validate the efficacy of our proposed model, we employ three datasets: MSL, SMAP, and ETL. MSL and SMAP are publicly available datasets released by NASA (O’Neill et al., 2010) and have gained widespread utilization in anomaly detection tasks related to multivariate time series data. Additionally, the ETL dataset was collected through randomly and repeatedly executing 60 distinct ETL tasks via scripting in a commercial company’s ETL system. The performance metrics dataset, which encompassed the utilization of critical hardware resources such as CPU, memory, and disk I/O, was gathered using the Prometheus tool. The overall duration of this dataset spans 104.5 h, with a collection frequency of once every five seconds. A detailed statistical overview of these three datasets is presented in Table 1.

• Evaluation metrics: We utilize precision, recall, and F1-Score as evaluation metrics to assess the model’s performance. Previous works, such as those by Tian, Su & Yin (2022), Zhao et al. (2020), and Tuli, Casale & Jennings (2022), often adopt a lenient approach, termed “soft identification,” where the entire abnormal interval is deemed correctly identified if only one point within it is marked as abnormal. However, this method can inflate the precision and recall scores, particularly when dealing with long abnormal intervals in the test set, leading to a potentially misleading performance evaluation. To address this limitation, we introduce a more stringent evaluation strategy known as “hard identification.” Under this approach, the identification status of other points within an interval remains unaffected, even if the interval itself is marked as abnormal. This approach ensures a more accurate assessment of the model’s true performance.

Experimental setup

Experimental results

Tables 3 and 4 present the performance metrics of each model, utilizing the “soft identification” and “hard identification” methods, respectively. Given the varying threshold selection mechanisms among methods, this study meticulously tests each model’s potential thresholds and reports the outcomes with the highest F1-Score. The tables present F1-Scores, emphasizing the peak value in bold and the second-highest in underline. In the soft identification approach, Table 3 reveals that EDD achieves superior performance on both the SMAP and ETL datasets, surpassing the second-best score by 3% and 7%, respectively. For the MSL dataset, it ranks second in F1-Score performance. Transitioning to the hard identification approach, as shown in Table 4, our proposed method surpasses all benchmark models across all datasets, with particularly impressive gains observed on the MSL and ETL datasets. Specifically, our method outperforms the second-ranked approach by 36% and 97% on the MSL and ETL datasets, respectively. A comparison of the two tables highlights the limitations of the evaluation approach used for previous models. Notably, several models that exhibit robust performance in the soft identification approach display subpar performance in the hard identification approach. This discrepancy is attributed to the soft identification evaluation approach, which conceals incorrect detections within a specific range if a model identifies an anomaly within that range. Conversely, the hard identification approach does not allow for such concealment, leading to a significant decline in the model’s performance metrics.

Ablation studies

To assess the effectiveness of each component in our EDD model, we conducted a series of ablation experiments. Specifically, we excluded the key configurations of each model component individually to observe the corresponding performance changes. To ensure rigorous evaluation, we utilized the F1-Score metric from the hard identification approach, which offers stricter criteria for performance assessment. This allowed us to precisely detect any performance degradation in the model.

In the first experiment, we disabled the reconstruction loss (Loss_rec) to explore the impact of the Encoder module on model performance. This variant was designated as EDD-REC. Subsequently, we turned off the KL loss (Loss_KL) to investigate whether clustering normal data effectively enhances model performance. This variant was labeled as EDD-KL. Lastly, we eliminated the negative loss (Loss_neg) to assess the guidance provided by abnormal data on the model. This variant was denoted as EDD-NEG.

The experimental results are presented in Fig. 2. These findings indicate that each key module in our model contributes positively to improving overall performance. Notably, the introduction of a limited amount of abnormal data had the most significant impact on enhancing model performance. This underscores the critical role of abnormal data in guiding the model to identify anomalous patterns effectively.

Data distinguishability

In anomaly detection, algorithms typically aim to establish a threshold that effectively separates abnormal data from normal data, solely relying on anomaly scores. This approach disregards the distinctiveness between normal and abnormal data. However, a model that demonstrates superior discrimination between these two categories is likely to excel in tasks that demand greater distinctiveness, thereby indicating its overall superior performance. To compare our approach with others, we conducted studies using the highly effective deep learning method, MTAD-GAT.

To visualize the anomaly scores on the test set, we generated box plots, as shown in Fig. 3. Specifically, subfigure A, B, and C represent the box plots for the MSL, SMAP, and ETL datasets respectively. The figures clearly illustrate that the proposed EDD method exhibits a more pronounced distinction between normal and abnormal data compared to MTAD-GAT. This enhanced distinguishability is crucial for accurate anomaly detection.

Additionally, while analyzing the ETL dataset (Fig. 3C), we observed that MTAD-GAT did not exhibit clear distinguishability between normal and abnormal data. Despite this, the F1-Score, computed using the soft identification evaluation method, surprisingly reached a remarkably high score of 0.8372. This apparent paradox further validates the concern raised in this article: that the soft identification evaluation approach may not always provide an accurate assessment of model performance. Therefore, this article introduces a hard identification evaluation approach to ensure a more precise performance evaluation of the model.

Latent space distribution

The idea of implementing anomaly detection in this article is to distinguish normal data from abnormal data based on their different probability distributions in the latent space. In the Encoder of the EDD model, we use the Loss_KL to map the distribution of normal data in the latent space to a normal distribution, while making no constraints on the distribution of abnormal data. This allows the normal data to be more clustered in the latent space, and the abnormal data to be randomly distributed. The Discriminator only needs to learn the difference between normal data and a small amount of abnormal data to distinguish most unseen anomalies. We visualized the encoding in the latent space on a 2D plane, as shown in Fig. 4. The visualization results are consistent with our expectations. The normal data (green points) are clustered at one point, and the model’s seen abnormal data (yellow crosses) and unseen abnormal data (purple rectangles) are randomly distributed in the latent space.

Anomaly proportion

The core of the algorithm in this article lies in optimizing the model’s ability to detect anomalies through a small amount of anomalous data. However, in real-world applications, the scarcity and difficulty of acquiring anomalous labels have become crucial factors restricting the improvement of model performance. To deeply explore the specific impact of the proportion of anomalous data on model performance, we designed an anomalous data ratio parameter γ in the experiment, which adjusts the proportion of anomalous data extracted from the test set for training to the total amount of anomalous data. The experimental results, as shown in Fig. 5, clearly demonstrate the model’s F1-Score performance under different anomalous data ratios.

The figure shows that in the absence of anomalous data for training, the model’s performance on numerous datasets is poor, emphasizing the importance of anomalous data in model training. However, by incorporating a modest bit of aberrant data as training guidance, the model’s performance improves dramatically. Remarkably, with only 1% of anomalous data included, the model’s F1-Score increased by more than three times on average. This conclusion emphasizes the relevance of crucial information contained in anomalous data for improving the model’s anomaly detection capacity, as well as providing useful insights into how to successfully employ limited anomalous data in actual applications. Furthermore, the experimental results also show that even with a very low anomalous data ratio (around 5%), the model can still effectively improve its anomaly recognition performance. This finding further validates the feasibility of deploying the algorithm in practical applications.