Automating Microservices Test Failure Analysis using Kubernetes Cluster Logs

Ericsson, a leading Information and Communication Technology (ICT) service provider, is at the forefront of pioneering cloud RAN¹, a technology that handles network traffic in the cloud. To increase development efficiency, Ericsson developed the Application Development Platform (ADP) ecosystem [4], which among other things, includes a marketplace that hosts over 280 microservices. Among these, 50+ microservices are common microservices that can be reused and integrated across all applications within the organization. The microservices in the ADP ecosystem are hosted in Docker containers and managed by Kubernetes.

Some applications need a specific set of microservices that may differ from others. In addition, the applications may deploy microservices in different environments, such as Amazon Web Service (AWS), or Microsoft Azure Cloud. To ensure high quality, the ADP ecosystem includes a Continuous Integration and Continuous Deployment (CICD) team, which simulates the different application environments and performs various tests, including testing scalability, robustness, and resilience.

Whenever a test fails, the developers must classify the failure into the following categories - an issue in the cluster, artifactory, microservice, CICD tests itself, or environment. The classification is done to report the failure to the relevant team. For example, microservice bugs are assigned to the relevant microservice team, while CICD bugs are reported to the CICD team for resolution. In addition, knowing the failure reason can provide Ericsson insights on what they should improve. These tests generate an average of 450 MB of log data. Developers use their knowledge and experience to classify the failure, which may take up to two hours for initial analysis. As the number of microservices and teams using them grows, it becomes increasingly challenging and time-consuming for developers to classify these failures manually.

This paper reports the experience of automating the classification of failures using machine learning techniques at Ericsson. The goal of automation is to identify an efficient and cost-effective method to decrease the time required for manual analysis. To achieve this goal, we utilized developers’ knowledge and experiences to better understand the workflows and information necessary for classifying the failure. Based on this understanding, we extracted important log data to train a machine-learning model capable of accurately predicting the failure's cause.

Figure 1 represents the steps followed in this study. In this section, we describe the process followed in each step.

Sequence Flow diagram

During the interviews, developers scrutinized a failed test case and analyzed the Kubernetes cluster logs to classify the failure. Specifically, they searched for logs associated with the microservice under test within the pods/container or pods/describe directories. If no relevant logs were found, the logs of dependent microservices were consulted. Our analysis revealed that the error or relevant information could usually be found in these directories. Even when it was not, there were clues indicating potential failure type. As a result, we concluded that logs located in pods/containers and pods/describe directories are crucial in most instances. To further strengthen this conclusion, we also examined the tickets submitted by the developers. We found that the root directories for 43 of the 50 reported log snippets were located in pods/container or pods/describe, or both directories.

The pods/containers and pods/describe directories were determined to be important during the interview phase. The logs were pre-processed, and the logs’ size was significantly reduced (up to 96.08%) through data selection and cleaning, reducing computational resources.

The classification algorithms were used to process the training data and generate a classification model, which was then used to predict the test data's class labels. The performance of the algorithms was evaluated based on their accuracy, F1-score, training time, and prediction time.

We compared the chosen classification algorithms statistically based on the recommendations by Demšar [1]. We applied the Friedman test to test the null-hypothesis that all algorithms are equivalent. If this hypothesis was rejected, we conducted a post-hoc Nemenyi test to compare the algorithms pairwise.

Table 1 presents the accuracy and F1-scores of each algorithm. The Friedman test was performed to evaluate the statistical significance of the differences between the algorithms, resulting in a p-value of 0.00074 and a Q value of 19.12, confirming significant differences in accuracy. We subsequently conducted a Nemenyi test, and the p-values of the combinations of algorithms are shown in Table 2 with P-values less than α (0.05) in bold. Random Forest and Gradient Boosting were statistically better than KNN.

Similarly, for F1-scores (Table 3), the Friedman test yielded a p-value of 0.00056 and a Q value of 19.76, indicating significant differences. The Nemenyi test showed that Random Forest outperformed SVM and KNN with statistically significant differences. Furthermore, KNN's performance was significantly lower than Gradient Boosting's. Regarding both F1-score and accuracy, Random Forest, Gradient Boosting, and MLP performed similarly.

We perform the Nemenyi test to determine which algorithms are statistically different based on F1-scores. Table 2 represents the p-value of F1-score for each combination of algorithms. The p-value in bold is less than α (0.05).

Finally, the algorithms were evaluated based on their training and prediction times. The average training time in minutes for all ten folds is shown in Table 4. It was observed that Gradient Boosting took a significantly longer time for training when compared to other algorithms. On the other hand, MLP took more than 2x longer for training compared to the Random Forest. Regarding prediction time, all five algorithms finished predictions in less than a second, indicating no significant difference between them in real-world performance. Based on the analysis, it can be concluded that the Random Forest algorithm performed better in terms of both accuracy and training time.

To test the model in a production environment, we developed an end-to-end prototype. The results of the model were added to a dashboard for developer analysis. We received feedback from developers, who indicated that while the failure classification helps them understand the reason for the failure and where they should look in the logs, they would appreciate additional information beyond the failure reason. As part of future work, we will devise an additional solution to include the most similar log previously reported along with its diagnosis report. Reviewing diagnosis reports of similar logs will help developers quickly understand the cause of failure. This way, developers can precisely look at the specific logs increasing their efficiency and helping them close the cases faster.

In addition, developers expressed concerns about the trustworthiness of the models’ predictions due to a lack of understanding of how the model arrived at its conclusions. Further effort is required to improve the models’ interpretability and explainability.

Although Random Forest performed better, we need to tune the hyperparameters to truly capture the effectiveness of Random Forest compared to other algorithms.

Identifying the reason for a failed CICD test is important to assign the failure to the relevant team for resolution. The automatic identification of failure reasons minimizes the manual effort needed to analyze the failure. In this study, we report experiences of classifying large-scale Kubernetes cluster logs using machine learning classification algorithms: Support Vector Machines (SVM), K-Nearest Neighbors (KNN), Random Forest Classifier, Gradient Boosting Classifier, and Multilayer Perceptron (MLP). Each log generates 450 MB of data on average, which was reduced to 91% by identifying relevant parts of the logs using expert opinion. We used the TF-IDF feature extraction method. Then we trained and evaluated the classification algorithms in terms of accuracy, F1 scores, training, and prediction time.

The results indicated that Random Forest outperforms SVM, KNN, Gradient Boosting, and MLP in terms of accuracy and computational cost. When evaluating the proof-of-concept, the developers expect additional information to strengthen the confidence in the prediction. For example, providing similar logs and diagnosis reports can help the developers to understand how previously failed tests were classified manually and arrived at the same classification predicted by the model. As part of future work, we aim to make the comparison of the algorithms more rigorous by applying hyperparameter tuning and further improving the model's interpretability and explainability.

As a prospect for future research, we intend to enhance the dimensionality of the input data. Currently, we concatenate logs from different services into a single file, creating a two-dimensional input. Our plan is to aggregate logs from different pods within each service, producing a three-dimensional input where each service has its own file. This approach would enable us to leverage deep learning algorithms, such as CNNs, which are proficient in processing 3D data. By doing so, our model would be able to identify the interdependencies among the services, leading to improved results.