Analysing Semi-Supervised ConvNet Model Performance with Computation Processes

Abstract

The rapid development of semi-supervised machine learning (SSML) algorithms has shown enhanced versatility, but pinpointing the primary influencing factors remains a challenge. Historically, deep neural networks (DNNs) have been used to underpin these algorithms, resulting in improved classification precision. This study aims to delve into the performance determinants of SSML models by employing post-hoc explainable artificial intelligence (XAI) methods. By analyzing the components of well-established SSML algorithms and comparing them to newer counterparts, this work redefines semi-supervised computation processes for both data preprocessing and classification. Integrating different types of DNNs, we evaluated the effects of parameter adjustments during training across varied labeled and unlabeled data proportions. Our analysis of 45 experiments showed a notable 8% drop in training loss and a 6.75% enhancement in learning precision when using the Shake-Shake26 classifier with the RemixMatch SSML algorithm. Additionally, our findings suggest a strong positive relationship between the amount of labeled data and training duration, indicating that more labeled data leads to extended training periods, which further influences parameter adjustments in learning processes.

1. Introduction

Recent advancements in semi-supervised machine learning (SSML) have led to the development of increasingly intricate models, architectures, and methods for parameter tuning, resulting in some of the most advanced algorithms to date [1]. However, as these algorithms become more sophisticated, so too does their complexity, which has led to a demand for greater transparency, particularly when the algorithms produce questionable results. When outcomes are inaccurate, it becomes even more essential to have clear explanations. Moreover, even with accurate results, the ability to trust in machine learning depends on being able to understand and justify how the decisions were made. This is crucial in high-stakes scenarios. As such, explainable artificial intelligence (XAI) methods are being developed to make the inner workings of these complex SSML algorithms more transparent, moving away from the notion of them as ‘black boxes’ to systems whose reasoning can be understood and trusted by users. There are two primary categories of XAI: post-hoc methods and model-based XAI. If an ML model’s design ensures user transparency regarding its internal operations and decision-making, it is classified under model-based XAI. Due to the profound intricacy of SSML, this approach is unsuitable. Hence, post-hoc XAI, which evaluates the decision-making logic of these enigmatic AI models, is critical, especially for grasping SSML algorithms’ overarching rationale.

Amidst XAI’s complexities [2], labeled data acquisition remains a challenge [3]. The ML community often combines a vast volume of unlabeled data with minimal labeled counterparts in standard SSML algorithms. These algorithms typically include a network classifier and two primary costs: classification and consistency costs. While the former utilizes labeled data, the latter assesses discrepancies between network classifier augmentations using unlabeled data. Although familiar SSML algorithms incorporate designs such as the $\Pi$ model [4] newer algorithms such as MixMatch and RemixMatch employ innovative patterns enhancing performance metrics.

DNNs as network classifiers have amplified learning accuracy [5]. Still, their intricacies often shroud performance justifications, leading to arbitrary adjustments in custom SSML algorithms. Recognizing this, our study systematically investigates SSML algorithm variations, focusing on learning accuracy and training effects related to DNN and SSML integration. Using prior frameworks [6], we unveil the fundamental factors influencing learning and training. Moreover, we employ XAI methodologies to devise a preprocessing-focused semi-supervised computation process (PF-SSCP), which essentially mirrors previous designs with slight reordering.

By integrating contemporary DNNs, including Shake-Shake26, DenseNet [7], and WideResNet [8], with each SSML algorithm, we establish a benchmark. Preliminary results highlight the performance superiority of Shake-Shake26 over DenseNet by 1.13% in terms of loss. However, this performance comes at the cost of extended training durations and increased resource consumption due to intricate interlayer connections. This raises our study’s central question:

Which are the most influential factors impacting computational costs and training accuracy when melding deep neural networks (DNN) with semi-supervised machine learning (SSML)?

Our research contributes in the following ways:

• We propose a framework method for connecting contemporary DNNs with SSML and illustrate a computation process with highlights of the most critical functions.
• We outline essential preprocessing steps when implementing these networks across varying labeled/unlabeled data ratios.
• We synergize the preprocessing-focused SSCP (PF-SSCP) with the classifier-focused SSCP (CF-SSCP), offering insights to maximize performance.
• We present SSML algorithms’ anatomy, furnishing guidelines for future model design rooted in this anatomy’s understanding.

2. Related Work

Several approaches exist for explaining CNNs, with one of the most notable being the unification and taxonomic view of graph neural networks (GNNs) [9]. This significant work delineates the commonalities and distinctions among existing methodologies, offering a foundation for future advancements [10]. The evaluation is streamlined by creating a suite of benchmark graph datasets specifically for GNN explainability. This encompasses metrics and datasets essential for appraising GNN explainability [11]. Nonetheless, complexities escalate when these concepts are applied to SSML algorithms due to the employment of multiple DNNs as network classifiers. Our approach delves into model-level explanations, directing attention to the taxonomy of SSMLs, and underscores precise changes and their driving forces.

An alternative study presented an interpretable compositional CNN approach [12,13], aiming to refine the conventional convolutional neural network (CNN) structure. This strategy is designed to facilitate the learning of filters that capture meaningful visual motifs within intermediate convolutional strata. While effective for illustrating a singular CNN, its applicability falters with SSML, where the network classifier integrates two or more CNNs.

Additionally, the XGNN method, which is a novel approach to graph generation, trains a graph generator to approximate predictions for target graphs rather than optimizing the input graph directly [14]. The graphs generated by this method are considered to be accurate explanatory targets that capture the unique patterns characteristic of graphs. The XGNN’s graph generation process uses reinforcement learning, allowing it to incorporate any competent graph generation algorithm within its framework, which broadens the scope and applicability of its explanations. This approach enhances the overall understanding of how trained graph neural networks (GNNs) function. However, while XGNN demonstrates strong performance in explaining graph classification models, its application to SSML algorithms is less effective. To address this, our technique leverages the robust graph generation capabilities of XGNN but enhances them by adding a layer of decomposition based on instance-level explanations [15]. This serves to shed light on three critical aspects of the latest SSML algorithms, providing a more nuanced understanding.

Recent advancements in SSML extensively use certain techniques, including consistency regularization, self-training, and entropy minimization. The study reported in [16,17] indicates that SSML methods may experience a significant decline in performance when there is a discrepancy between the distribution of unlabeled data used during training and those encountered post-training, even if other variables such as initialization, preprocessing, regularization, and augmentation are well-managed. Conversely, our research focuses on developing a generalized approach to these techniques and elucidating the specific procedural steps that lead to better performance, independent of the specific characteristics of the unlabeled data.

Informed machine learning [16] integrates supplementary prior knowledge during model training. The referenced study provides a comprehensive synopsis of diverse strategies within this domain, meticulously outlining the informed machine learning design process and its numerous constituent elements. It also clearly distinguishes informed ML from conventional ML paradigms. Our research, however, zeroes in on SSML algorithms, starting with the initial design, then delving deeper into the taxonomy of each and revealing the influential factors in every segment.

3. Survey Analysis of SOTA Models with Modular Computation Components

This section conducts a technical survey of SOTA semi-supervised learning (SSL) models, focusing on dissecting and visualizing the computational components to examine their modularity. The purpose of this modular-based analysis is to abstract away from the intricate parameter-level specifics of each model and instead to concentrate on their functional attributes. The goal is to distill a computation process that encapsulates the shared computational components found within various models. Through this high-level analysis, we aim to pinpoint critical computational junctures that may be leveraged to enhance learning performance. The models under consideration, which include Temporal Ensembling, the $\Pi$ model, Mean Teacher, MixMatch [18], and ReMixMatch [19], are explored for their design, taxonomy, and innovative contributions to the field of SSML. We intend to illustrate the interplay of these advanced techniques with the proposed semi-supervised computation process (SSCP), tracing the connections between modular computation components and the original models’ equations.

3.1. Temporal Ensembling

The Temporal Ensembling structure [4] maintains a configuration that can be depicted in Figure 1, featuring a network classifier with a single DNN. The primary distinction between Temporal Ensembling and the generic form with multiple DNNs lies in the fact that, here, network augmentations are assessed only once per epoch, and network consistency relies on previous evaluations for the unsupervised loss component.

In the Temporal Ensembling model, the data $x_i$ undergo the same preprocessing steps to achieve optimal performance. After this phase, the preprocessed data are fed into a neural classifier network, which determines the classification prediction $z_i$. In the subsequent iteration, $z_i$ is aggregated, aiding in the computation of both consistency and classification costs in the current iteration. This mirrors the process followed in generic SSML; however, the computation of the consistency cost is exclusive to the unlabeled data. In Figure 1, the critical path for various steps, in the case of unlabeled data, is illustrated as a black path. For labeled data, both classification and consistency costs are calculated, and the forward propagation is depicted by the black and orange paths. Moreover, the total loss computation is articulated in Equation (12), mirroring that of the generic SSML.

The classification cost computation aligns precisely with that of generic SSML, as detailed in Equation (11), and is defined as the cross-entropy between the prediction $z_i$ and the label $y_i$.

Regarding the consistency cost, despite being governed by the same Equation (10), there is significant variation in the target vector $\widetilde{z_i}$. Here, $\widetilde{z_i}$ is not derived from another neural network’s assessment under varying dropout conditions. Instead, it is an accumulation of ensemble predictions preceded by a bias correction step. The neural network computes $z_i$, which is then amalgamated into the ensemble outputs $Z_i$ following the update $Z_i\leftarrow \alpha Z_i+(1-\alpha )z_i$ [4], with $\alpha$ representing the momentum term that dictates the depth of historical training data influencing the ensemble. During the initial iteration ($i=0$), $z_{i-1}$ is set to zero, indicating that the initial values for $\tilde{z}$ and Z are also zero. Consequently, it is imperative to correct the startup bias in Z by dividing it by the factor $(1-{\alpha }^t)$ to obtain the training target ${\tilde{z}}_i$. In the context of backward propagation, this loss is applied solely to the current neural network.

Temporal Ensembling offers several advantages over other SSML algorithms, as outlined below:

• Since the network undergoes evaluation once for each input, the training process is expedited.
• With a training target characterized by reduced noise compared to the $\Pi$ model, Temporal Ensembling can potentially realize superior accuracy performance.

However, Temporal Ensembling also presents certain limitations, detailed as follows:

• It necessitates the storage of auxiliary data and the introduction of a new hyperparameter, $\alpha$, thereby demanding additional memory resources.
• Similar to the $\Pi$ model, the Temporal Ensembling model lacks dynamic learning capabilities, necessitating system retraining for effective functionality.

3.2. $\Pi$ Model

This model, a variant of SSML, employs a self-ensembling mechanism during training [4], ensuring network output consistency across two instances of the same input under different dropout conditions.

In Figure 2, both labeled and unlabeled data, $x_i$, are subjected to identical preprocessing, without discrepancy. Upon entry into the network classifier, the data are replicated across two analogous neural networks. These networks are differentiated by the following factors:

• Dropout regularization, infusing an element of randomness into these twin architectures.
• The types of augmentations employed, which expand the dataset and render identical labeled and unlabeled data as distinct inputs.

These two neural networks yield the classifications $z_i$ and ${\tilde{z}}_i$, which are instrumental in computing the total loss. When labeled data are input into the network classifier, both consistency and classification costs are derived. As shown in Figure 2, this scenario necessitates following the black and orange paths for forward propagation to ascertain the total loss.

Conversely, for unlabeled data, only the consistency cost is derived, meaning that the black path in Figure 2 is pursued during forward propagation. Ultimately, the losses across the entire dataset are aggregated via a weighted sum. A pivotal aspect here is the employment of the ramp-up function $w\left(t\right)$, which assigns weights to the consistency costs, compelling the semi-supervised system to prioritize classification costs in the initial stages. Throughout the loss computation, a backward propagation must be initiated to update the parameters aimed at loss reduction. In Figure 2, backpropagation is signified by the red path, indicating that both neural network parameters, $\theta$ and ${\theta }^{\prime }$, are updated, thus revising the entire neural network flow.

Evaluating the performance of the $\Pi$ model, we observe that it offers better accuracy than preceding SSML algorithms. However, this model has the following limitations:

• The network output is computed twice for the same input data, resulting in additional computational costs.
• The model is vulnerable to errors when fed incorrectly labeled data, negatively impacting its accuracy.
• This model cannot learn dynamically as it relies on batch-learning, in contrast with online learning neural networks.

3.3. Mean Teacher

To address the shortcomings of the two SSMLs given in the previous subsections, the Mean Teacher method calculates the averages of the model weights rather than predicting the outcome. This model is based on an average of consecutive student models; hence, it is referred to as the Mean Teacher method [4].

In this case, the raw data $x_i$ are first preprocessed and then injected into the network classifier in the same manner as the previous SSML models. However, in this case, the network classifier is not the same as previous models. Here, the network classifier consists of two identical DNNs with dropout conditions referred to as the student and teacher models. Primarily, the Mean Teacher model is exactly the same as the $\Pi$ model during the computation of forward propagation for both unlabeled and labeled data. However, the Mean Teacher method diverges from the $\Pi$ Model during the backward propagation phase and in the strategy it employs to update weights.

The backward propagation is simply computed for the the parameters $\theta$ of the student model. For the teacher model, the parameter update for $\tilde{\theta }$ happens using the parameters of the student model, $\theta$, by calculating the exponential moving average (EMA) as shown in Figure 3. This can be observed in Equation (1) [4]:

$${\tilde{\theta }}_t=\alpha {\tilde{\theta }}_{t-1}+(1-\alpha ){\theta }_t$$

(1)

where

• ${\tilde{\theta }}_t$ specifies the parent weights;
• ${\tilde{\theta }}_{t-1}$ signifies the parent weights of the previous run;
• ${\theta }_t$ specifies the child weights; and
• $\alpha$ specifies the smoothing parameter.

It has been observed that averaging the model weights over various training steps is more likely to provide a more accurate model compared to utilizing the final weights promptly. In the case of the Mean Teacher model, the information can be aggregated after each step rather than every epoch. However, previous research studies have overlooked a significant metric, which is the difference in the program’s run-time. It has been observed that the Mean Teacher model can be trained in much less time than the $\Pi$ model, and it achieves much improved accuracy compared to Temporal Ensembling. However, there are some drawbacks of the Mean Teacher model:

• Both the supervised networks perform forward propagation using different parameters. However, the student model is the only one with backward propagation to compute the gradients.
• The Mean Teacher model utilizes much greater memory resources compared to the other two models since the dropout conditions ($\eta$ and ${\eta }^{\prime }$) must be kept and preserved during each epoch.

3.4. MixMatch

Compared to the previous methods discussed, MixMatch [18] has a different approach and follows the PF-SSCP method. In every batch, this model augments the raw labeled data $x_b$ and keeps the labels $y_b$, generating the dataset $\widehat{X}$. For the unlabeled data, the raw input $x_b$ is augmented K times, where K is a hyper-parameter. Then, those K augmented entries are passed through the network classifier to predict all their labels (${\tilde{z}}_{b2},{\tilde{z}}_{b2}$ to ${\tilde{z}}_{bk}$), and after that, their average, ${\tilde{z}}_{bavg}$, is taken as a prediction for all the K entries. ${\tilde{z}}_{bavg}$ is sharpened using temperature sharpening to reduce the entropy of the label distribution. This can be carried out as follows:

$$\begin{array}{c}\hfill \mathrm{Sharpen}{(p,T)}_i:=\frac{p_i^{\frac{1}{T}}}{{\sum }_{j=1}^L{p}_j^{\frac{1}{T}}}\end{array}$$

(2)

where

• p is some input categorical distribution;
• T is a hyperparameter that needs to be tuned; and
• L is the number of labeled classes.

As such, the output of the sharpening will produce unlabeled data with the pseudo-labels ${\overline{z}}_b$, denoted as $\widehat{U}$. Concatenating and shuffling both $\widehat{X}$ and $\widehat{U}$, we obtain W. MixUp is conducted on $\widehat{X}$ and the first $\left|X\right|$ entries of W to obtain $X^{\prime }$, where $\left|X\right|$ is the size of the labeled data in the batch. Additionally, MixUp is applied on the unlabeled data in the batch $\widehat{U}$ with the rest of the entries W to obtain $U^{\prime }$. To understand how MixUp works, we need to understand how $(x^{\prime },p^{\prime })$ is computed using two examples with their corresponding labels and probabilities, $(x_1,p_1),(x_2,p_2)$, as shown below:

$$\begin{array}{c}\hfill \lambda \sim \mathrm{Beta}(\alpha ,\alpha )\end{array}$$

(3)

$$\begin{array}{c}\hfill {\lambda }^{\prime }=\max (\lambda ,1-\lambda )\end{array}$$

(4)

$$\begin{array}{c}\hfill x^{\prime }=\lambda x_1+(1+{\lambda }^{\prime })x_2\end{array}$$

(5)

$$\begin{array}{c}\hfill p^{\prime }=\lambda p_1+(1+{\lambda }^{\prime })p_2\end{array}$$

(6)

where $\alpha$ is the hyperparameter to tune. Due to $\lambda \ge 0.5$ and the $max$ function, the first term $x_1$ and its label $p_1$ have more importance than the second point $x_2$ and its label $p_2$. This makes the MixUp model’s prediction for $X^{\prime }$ and $U^{\prime }$ correspond to labeled and unlabeled output guesses, respectively. As we know the exact output of labeled data, the classification cost is basically the cross-entropy. On the other hand, for the consistency cost, since we are guessing the labels of the unlabeled data, the $L_2$ loss [20] is used. These losses can be shown in the equations below:

$$\begin{array}{c}\hfill L_X=\frac{1}{|X^{\prime }|}\sum _{x,p\in X^{\prime }}H(p,p_{model}\left(y\right|x;\theta ))\end{array}$$

(7)

$$\begin{array}{c}\hfill L_U=\frac{1}{L|U^{\prime }|}\sum _{x,p\in U^{\prime }}\left|\right|{\overline{z}}_b-p_{model}{\left(y\right|u;\theta )\left)\right||}_2^2\end{array}$$

(8)

$$\begin{array}{c}\hfill L=L_X+{\lambda }_U{L}_U\end{array}$$

(9)

3.5. ReMixMatch

Following the same principle as MixMatch, ReMixMatch [19] adopts the PF-SSCP method but introduces several modifications. One significant alteration is the augmentation anchoring method. As depicted in Figure 4, the labeled input $x_b$ undergoes a robust augmentation process using a variant of AutoAugment known as CTAugment, which concurrently learns an augmentation policy during model training. This strongly augmented labeled data ${\widehat{x}}_b$, together with the label $y_b$, constitutes the dataset $\widehat{X}$. For the unlabeled input $x_b$, the model employs multiple strong augmentations K, resulting in ${\widehat{x}}_{b,1..k}$. Using the same unlabeled raw input $x_b$, a weakly augmented version is generated by applying only a crop or a flip, yielding ${\widehat{X}}_{bw}$. This ${\widehat{X}}_{bw}$ is then processed through the network classifier to obtain the prediction ${\tilde{z}}_{bw}$.

Another notable modification is the implementation of distribution alignment on ${\tilde{z}}_{bw}$. This adjustment involves maintaining a running average of the network classifier’s predictions for the unlabeled data, denoted as $\tilde{p}\left(y\right)$, and estimating the marginal class distribution $p\left(y\right)$ for the labeled examples during training. The model then applies a ratio of $p\left(y\right)/\tilde{p}\left(y\right)$ to the network classifier’s prediction of the label $z=p_{model}\left(y\right|u;\theta )$, followed by normalization using $\mathrm{Normalize}{\left(x\right)}_i=x_i/{\sum }_j{x}_j$. Subsequently, temperature sharpening, as previously discussed in the context of MixMatch, is applied to produce the pseudo-labels ${\overline{z}}_{bw}$.

Utilizing both ${\widehat{x}}_{b,1..k}$ and ${\overline{z}}_{bw}$, the model generates the unlabeled dataset $\widehat{U}$. It is evident that the multiple strongly augmented versions of the unlabeled data are based on the weakly augmented variants. Beyond this stage, the procedure aligns precisely with that of MixMatch, adhering more generally to the PF-SSCP framework.

4. SSML Model Analysis Framework

As with many other ML algorithms, SSML algorithms maintain a similar structure, albeit with minor differences. Referring to Figure 5, we observe the network classifier-focused structure of previous SSML models, such as the $\Pi$ model, Mean Teacher, and Temporal Ensembling. The preprocessing of labeled and unlabeled data is conducted, which are then injected into the network classifier without splitting, simultaneously. Throughout the preprocessing step, data quality is enhanced by suppressing undesirable distortions or enhancing certain image features essential for further processing. For instance, some of these operations could include filtering, geometric transformations, pixel brightness adjustments, or a combination thereof.

Analyzing these SSML algorithms, the network classifier block comprises single or multiple instances of essentially the same DNN to execute the optimal data classification. Consequently, this unique network classifier design facilitates transformative parameter updates. This feature differentiates various SSML models. A network classifier can include the following types:

• One DNN: Here, a single DNN is employed in the classifier, conducting both backward and forward propagations.
• Two DNNs: This configuration utilizes two nearly identical DNNs, differing slightly in their dropout conditions and parameter initialization. Forward propagation occurs in both DNNs; however, backward propagation might not occur in one of these DNNs in some designs.
• Three or more DNNs: Here, three or more DNNs are employed, potentially with varying initialization conditions. Additionally, dropout conditions may or may not be integrated into these DNNs.

Depending on the total number of DNNs, the network classifier’s output integrates one or more classifications for computing the loss. Based on the data type input, there are two distinct cases for loss computation:

• Unlabeled data: For unlabeled data, the consistency cost among two or more DNN instances is calculated, multiplied by the number of classes and the inverse of the batch size.
• Labeled data: In this scenario, the consistency cost among two or more DNN instances and the classification cost between one or more models are computed.

It is noteworthy that the classification cost is calculated only for labeled data. For unlabeled data, the classification cost is not computed and is set to zero.

Upon assessing the most recent state-of-the-art MixMatch and ReMixMatch, we noticed a deviation in the structure of the previous semi-supervised computation process. These alterations are depicted in Figure 6.

As illustrated, preprocessing is applied to both unlabeled and labeled datasets, subsequently outputting postprocessed unlabeled and labeled datasets. The preprocessing step generates pseudo-labels for the unlabeled datasets, effectively rendering both datasets labeled [21]. Regarding the network classifier, in most designs, it is simplified to one DNN; however, there is potential for further exploration. The network classifier’s predictions are utilized for calculating the classification cost and the consistency cost. Examining each SSCP, the primary distinctions between PF-SSCP and CF-SSCP are as follows:

• The design’s emphasis is placed on preprocessing steps rather than the network classifier.
• The classification cost is consistently calculated due to the pseudo-labels on the unlabeled datasets.

While these established differences might seem minor, when evaluating these distinctions and conducting experiments on these SSML algorithms, the following research question was prompted:

RQ1: How do PF-SSCP compare to CF-SSCP in parameter updates in network classifiers and preprocessing techniques, respectively, measured in training time, learning accuracy, and training loss?

Furthermore, the consistency cost (also known as the unsupervised loss) can be computed using the classification outputs of one or more DNN instances, denoted as $z_i$ and ${\tilde{z}}_i$. Moreover, in instances where only one DNN is employed, ${\tilde{z}}_i$ represents the evaluations of the previous network, not a second network’s evaluation. Generally, the consistency cost is formulated as follows [4,22]:

$$\begin{array}{cc}\hfill \mathrm{Consistency}\mathrm{Cost}& =w\left(t\right)\ast \mathrm{Squared}\mathrm{Difference}\hfill \\ \hfill & =w\left(t\right)\ast \frac{1}{C\left|B\right|}\sum _{i\in B}{∥z_i-{\tilde{z}}_i∥}^2\hfill \end{array}$$

(10)

where

• $z_i$ represents $f_{\theta }\left(g\left(x_{i\in B}\right)\right)$ and $\theta$ denotes the function’s training parameter.
• ${\tilde{z}}_i$ denotes $f_{{\theta }^{\prime }}\left(g\left(x_{iinB}\right)\right)$, and ${\theta }^{\prime }$ represents the function’s training parameter.
• $\left|B\right|$ is the batch size.
• $w\left(t\right)$ signifies the time-dependent weighting function (discussed subsequently).
• C represents the number of classes, with $y_i\in \{1\dots C\}$ (e.g., cat, tree, dog, car, etc., for CIFAR-10).

When data are labeled, $z_i$ is used in conjunction with the label $y_i$ to compute the classification cost (or the supervised loss). The supervised loss, or classification cost, is essentially the log of the product of classification and the corresponding label. For mini-batches of size B, this is the inverse of the mini-batch size multiplied by the sum of the losses’ negative values, as shown in Equation (11) [4]:

$$\begin{array}{cc}\hfill \mathrm{Classification}\mathrm{Loss}& =-\frac{1}{\left|B\right|}\sum log{z}_i\left[y_i\right]\hfill \end{array}$$

(11)

Combining both losses provides the total loss used for updating the network’s parameters. In light of these computations and the observed variations in SSCP designs, the following research question emerges:

$$\begin{array}{cc}\hfill \mathrm{Loss}& =\mathrm{Classification}\mathrm{Cost}+\mathrm{Consistency}\mathrm{Cost}\hfill \\ \hfill & =\mathrm{Cross}\mathrm{Entropy}+w\left(t\right)\ast \mathrm{Squared}\mathrm{Difference}\hfill \\ \hfill & =-\frac{1}{\left|B\right|}\sum _{i\in (B\cap L)}{y}_i\ast log{z}_i+w\left(t\right)\ast \frac{1}{C\left|B\right|}\sum _{i\in B}{∥z_i-{\tilde{z}}_i∥}^2\hfill \end{array}$$

(12)

where

• $z_i$ specifies the $f_{\theta }\left(g\left(x_{i\in B}\right)\right)$ and $\theta$ specifies the function training parameter.
• $\left|B\right|$ represents the batch size.
• L includes the labeled input indices.
• $y_i$ is the label for input $x_i$.
• $w\left(t\right)$ specifies the time-dependent weighting function (discussed later).
• C specifies the number of different classes $y_i\in \{1\dots C\}$ (cat, tree, dogs, car, etc..)

As the loss has been decomposed, and it highly depends on the labeled/unlabeled data ratio, it raised another subquestion during our experiments:

RQ2: Given the consistency and classification costs’ computation methods in PF-SSCP and CF-SSCP, how do these costs impact the effectiveness of the semi-supervised learning models in terms of accuracy and loss during the training and inference phases?

The research questions RQ1 and RQ2 delve deep into the comparative analysis of the PF-SSCP and CF-SSCP methods, focusing primarily on their structural differences and the implications these hold on the effectiveness of semi-supervised machine learning models.

5. Empirical Evaluation Method

Before diving into the experiments and assigning values to specific hyperparameters, it is important to discuss the methodology used to address the research questions. The adoption of various techniques for applying XAI methods [3,23] to analyze trained models has steered XAI towards post-hoc analysis. It is essential to remember that the complexity of a machine learning model is often inversely related to its interpretability [24]. Generally speaking, the more complicated and unconstrained a model is, the more difficult it is to interpret or explain clearly [25]. Thus, integrating CNNs within SSML algorithms results in non-monotonic and non-linear response outputs, contributing to the creation of models that are among the least interpretable. In this context, our methodology is model-specific, concentrating on global measures that will enable us to comprehend fully the inputs and their complex modeled correlations with the output predictions.

5.1. Datasets

In this section, we present the dataset utilized for training semi-supervised machine learning (SSML) classifiers. The CIFAR-10 dataset was selected, comprising 60,000 distinct 32 × 32 color images evenly distributed across ten different classes, with each class containing 6000 images. From this dataset, we allocated 10,000 images for the testing process and the remaining 50,000 images for the training phase.

In terms of the dataset, CIFAR-10 was selected for its simplicity and its common use in benchmarking the chosen state-of-the-art SSML classifiers. We reduced the labeled data in various ratios primarily to address RQ2 while also evaluating the comparison of PF-SSCP and CF-SSCP on aspects such as training time, training loss, and learning accuracy, thereby contributing to the resolution of RQ1. Given our objective of testing the SSML algorithms with varying quantities of labeled data, we intentionally withheld some of the labels during the training of these SSML algorithms, as detailed in

Table 1. CIFAR-10 Training Datasets.

Labeled Data	Unlabeled Data	Labeled Data per Class	Unlabeled Data per Class
1000	49,000	100	4900
4000	46,000	400	4600
50,000	0	5000	0

.

As depicted in Table 1, both labeled and unlabeled data are proportionately distributed across the ten classes, indicating an impartial approach with no class preference. The substantial gap between the 4000 and the 50,000 labeled datasets serves as an ideal measure for assessing the efficacy of the unsupervised components relative to the supervised elements within various SSML algorithms. In the case of the 1000 labeled datasets, the SSML algorithms undergo rigorous testing, primarily relying on the unsupervised components of the network, thereby approaching the realm of unsupervised machine learning algorithms [26].

5.2. Preprocessing

The preprocessing step primarily hinges on the specific SSML approach and the dataset in use. For the purposes of this paper, preprocessing is tailored to either the PF-SSCP or the CF-SSCP methodologies. In CF-SSCP, we execute zero component analysis (ZCA) [21], a step adopted by both the Mean Teacher and $\Pi$ models in their processing of the CIFAR-10 datasets, which contributes to enhanced accuracy and diminished loss. Conversely, preprocessing in the context of PF-SSCP, as demonstrated in strategies such as MixMatch and ReMixMatch, adopts a more intricate approach previously elaborated upon in Section 3.4 and Section 3.5. This method capitalizes on the network classifier’s capability to generate pseudo-labels, negating dependence on randomized algorithms. Additionally, a crucial element in both preprocessing-focused and classifier-focused SSCP is data augmentation, a strategy that significantly betters results by enriching the training dataset. We executed various data augmentation techniques, including image flips, zooms, shifts, and cropping.

5.3. SSML and DNN Combination

The choice of SSML model dictates the nature of integration to be employed, a necessity given the distinct behaviors exhibited by different architectural choices [27]. This combination evaluates five critical aspects:

• Performance, gauged through accuracy and loss metrics.
• The duration of the training process.
• Constraints imposed by hardware.
• The initial selection of hyperparameters.
• The appropriateness of the loss function.

Both PF-SSCP and CF-SSCP are guided by these metrics, though the degree of influence each one holds varies. This variance is attributable to specific design features outlined in the preceding section for each SSML type and their respective behaviors. In the context of CF-SSCP, most DNNs utilize dropout regularization, a technique instrumental in preventing network overfitting. Conversely, PF-SSCP does not mandate overfitting prevention measures within the DNNs.

5.4. Ramp-Up and Ramp-Down Functions

In CF-SSCP, a ramp-up period was initiated with 40,000 training steps at the onset of the training process. During this phase, both the learning rate and the consistency cost parameters were progressively increased from zero to their peak values using a sigmoid-shaped function. Conversely, for PF-SSCP, a linear ramp-up was employed, escalating from an initial value of 100 to the maximum over the initial 16,000 training steps.

5.5. SSML Performance Measurements

This segment details the approach adopted to assess the classification efficacy of the SSML networks, with a specific focus on the variety of network classifiers involved in the evaluation. As indicated earlier, our experimental analysis was confined to the CIFAR-10 dataset. Per the discussion in Section 4, any SSML algorithm can be abstracted into two distinct types of SSML architectures, as illustrated in Figure 5, by employing diverse network classifiers. In this study, we restricted our consideration to contemporary CNN models serving as the network classifiers in SSML networks. For the training and evaluation phases of the proposed systems, we utilized a GeForce GTX 2080 Ti GPU. This specific GPU was chosen for its proficiency in handling an array of SSML algorithm and network classifier combinations [28].

It is pertinent to mention that in CF-SSCP, for a streamlined analysis, we amalgamated the preprocessing stage with the feature selection process, as both these procedures precondition the datasets prior to their introduction to the network classifiers. In contrast, PF-SSCP placed a greater emphasis on the preprocessing stages as well as the generation of pseudo-labels, owing to the inherent design of PF-SSCP, to more significantly affect the dataset rather than the network classifier.

The ensuing performance metrics were established as the foundation for the appraisal of the proposed models:

• Performance accuracy and training loss, evaluated against varying proportions of unlabeled to labeled datasets.
• The duration of training necessitated by each amalgamation of SSML with diverse network classifiers, inclusive of the time expended during the preprocessing and feature selection stages.
• Utilization of parameters and the complexity intrinsic to various SSML algorithms, with an exposition on the distinct parameters requisite in each scenario and their subsequent influence on complexity.

5.6. Experimental Design and Framework Specifications

In order to rigorously evaluate the performance of semi-supervised learning models, our experiments incorporate two principal computational frameworks: the CF-SSCP and the PF-SSCP frameworks, as outlined in

Table 2. Experiments for different semi-supervised models under CF-SSCP and PF-SSCP frameworks.

Framework	Model	Experiment and Architecture	Summarized Purpose
CF-SSCP	Temporal Ensembling (TE)	Exp 1: TE and Shake-Shake26	Ensemble predictions over time for stability and consistency
		Exp 2: TE and DenseNet-121	Utilizing ensemble learning with a focus on network depth
		Exp 3: TE and WRN-40-2	Applying ensembles to widen and deepen network architectures
		Exp 4: TE and WRN-28-10	Ensemble methods combined with a wider network model
	$\Pi$ Model	Exp 5: $\Pi$ model and Shake-Shake26	Ensuring consistent network predictions without skip connections
		Exp 6: $\Pi$ model and DenseNet-121	Consistency of prediction with depth-oriented network architectures
		Exp 7: $\Pi$ model and WRN-40-2	Deeper networks under consistency constraints
		Exp 8: $\Pi$ model and WRN-28-10	Wider networks maintaining prediction consistency
	Mean Teacher (MT)	Exp 9: MT and Shake-Shake26	Teacher–student model consistency without skip connections
		Exp 10: MT and DenseNet-121	Depth and skip connections in a teacher–student setup
		Exp 11: MT and WRN-40-2	Deeper architecture in a mean teacher framework
		Exp 12: MT and WRN-28-10	Enhanced width in the teacher–student model’s architecture
PF-SSCP	MixMatch (MM)	Exp 13: MM and Shake-Shake26	Augmentation and mixing strategies for semi-supervised learning
		Exp 14: MM and DenseNet-121	Deep architecture applied to advanced mix-and-match techniques
		Exp 15: MM and WRN-40-2	Widening and deepening networks with semi-supervised mix–matching
		Exp 16: MM and WRN-28-10	Wide network structures in advanced mix–match learning scenarios
	ReMixMatch (RM)	Exp 17: RM and Shake-Shake26	Refinement of mix–match techniques with a preprocessing focus
		Exp 18: RM and DenseNet-121	Application of preprocessing strategies in deep learning models
		Exp 19: RM and WRN-40-2	Preprocessing alignment in wider and deeper network structures
		Exp 20: RM and WRN-28-10	Extensive preprocessing in a widened network scenario

. These frameworks provide the basis for comparing models by assessing the impact of classifier complexity and preprocessing sophistication. The table showcases the relevant experiments conducted, presenting a clear framework for analysis, while a comprehensive list of experimental hyperparameters and additional experimental details are reserved for the Appendix A section of this paper.

The selection of architectures for our experiments was deliberate to encompass a broad range of complexities and capacities pertinent to semi-supervised learning:

• DenseNet-121 (Dense Convolutional Network—121 layers): Known for its dense connectivity, DenseNet-121 optimizes parameter efficiency and facilitates feature propagation and reuse. Its design is particularly beneficial for learning with limited labeled data, which is a common challenge in semi-supervised learning scenarios [29].
• Shake-Shake Regularization Model (Shake-Shake26): The Shake-Shake regularization approach, exemplified by the Shake-Shake26 model, introduces stochasticity [30] into the training process. This method has been shown to enhance generalization on image classification tasks, presenting a unique advantage in semi-supervised learning frameworks [31].
• Wide Residual Networks (WRN-40-2): The WRN-40-2 architecture augments the network’s width, offering an optimal trade-off between depth and width. This expanded network capacity allows it to represent more complex functions and data relationships, benefiting from the additional unlabeled data in semi-supervised learning setups [8].
• WRN-28-10: The WRN-28-10 extends the width of traditional residual networks even further, targeting the rigorous demands of high-complexity classification tasks. The architecture is designed to capitalize on the unlabeled data that are more prevalent in semi-supervised learning contexts [8].

The adoption of these architectures in our experiments allows us to thoroughly assess the efficacy of semi-supervised learning strategies across varying levels of network complexity and depth.

6. Evaluation Results and Analysis

6.1. CF-SSCP with Various Network Classifiers

Given CF-SSCP’s substantial reliance on the network classifier, it becomes pertinent to investigate its performance dynamics across different network classifiers. Accordingly, our tests for CF-SSCP were conducted using both single (Mean Teacher and Temporal Ensembling) and double ($\Pi$ model) network classifiers. The ensuing subsections detail our findings.

6.1.1. Results of Temporal Ensembling

The Temporal Ensembling method, utilizing a singular DNN within the network classifier, stands as the most straightforward among the three SSML algorithms. As evidenced in Figure 7 and Figure 8, it is apparent that, under Temporal Ensembling, the efficacy of both WRN-40-2 and DenseNet-121 is overshadowed by the more robust Shake-Shake26 model.

Inspection of the accuracy data in Figure 9 reveals that training accuracy marginally surpasses test accuracy, with the highest accuracy observed at 4000 labels.

However, as Figure 10 demonstrates, the training loss for Shake-Shake26 within the Temporal Ensembling framework fails to reach optimal performance even with extensive hyperparameter tuning. This outcome likely stems from the model’s simplicity, concentrating solely on the training loss of a single DNN. Thus, exploration into the alternative models, namely the $\Pi$ model and Mean Teacher model, is warranted.

Summary:

In all three experiments, it is evident that Temporal Ensembling experiences the highest loss compared to the other two models, the $\Pi$ model and the Mean Teacher model. Analyzing the efficacy of various DNNs within the network classifier of Temporal Ensembling reveals that Shake-Shake26 incurs the least loss compared to the other two DNNs, namely, DenseNet and WideResNet. This suggests that a DNN with fewer skip connections can yield more accurate results and lower training loss. Additionally, employing two DNNs in a network classifier appears to facilitate better parameter updates during the computation of the consistency cost.

6.1.2. Results of $\Pi$ Model and Mean Teacher

We conducted training and validation for the $\Pi$ model and Mean Teacher model utilizing WideResNet as the primary network. This setup was designed to exhibit the performance metrics of each model by leveraging the CIFAR-10 dataset with varying percentages of labeled data.

• WideResNet Core Network with Mean Teacher: Figure 11 depicts the accuracy achieved by the Mean Teacher model using WideResNet as our core network. The graph indicates that the highest validation accuracy was achieved using the Stochastic Gradient Descent (SGD) [32] algorithm with 44,000 labels, the maximum in these experiments. Notably, accuracy diminishes as the label count decreases. This trend was anticipated since accuracy correlates with the total label count, assuming unchanged hyperparameters.
  However, the primary observation here concerns the discrepancy between the SGD and Adam optimizers. Clearly, the SGD optimizer [33] surpassed Adam in terms of implementation accuracy.
  Examining the loss in Figure 12, the Adam optimizer performs superiorly, with less test loss. Another notable aspect is both networks’ initial struggle to smoothly reduce loss, resulting in a highly noisy transition of the loss function. This is attributed to the unsuitability of maintaining consistent hyperparameter values across different volumes of labeled data.
  Another critical point pertains to initialization. Different initialization parameters are crucial, evident from the 10,000-label dataset starting significantly lower than the 44,000-label set, causing the latter higher initial loss and hindering performance.
• WideResNet core network with $\Pi$ model: For the Adam optimizer, accuracy does not directly correlate with the number of labels, as distinctly seen in Figure 11 and Figure 13.
  Here, high training accuracy is noted with 1000, 4000, and 44,000 labels. Among these, the 1000-label dataset provides the highest accuracy. However, the 10,000-label dataset’s training accuracy is 20% lower than the others, yet its testing accuracy, at 88.56%, surpasses the rest. Additionally, the 1000-label set displays an improvement in accuracy up to 58% before a decline, suggesting overfitting, as evidenced in Figure 14.
  Comparing this network with different label quantities, the loss is significantly lower at 10,000 labels, a result of hyperparameter tuning and network initialization. Even with 44,000 labels, the network struggles to reduce loss due to an unfavorable start.
  Moreover, the network’s commencement varies across datasets, except for the 10,000-label set, where loss increases from 0 to 30 epochs. Significant fluctuations are particularly noticeable for the 1000-label data. These issues could be addressed through hyperparameter tuning using strategies such as Random Search or Grid Search.
• Comparison of Mean Teacher with $\Pi$ model using WideResNet as the core network: Figure 15 compares the testing and validation accuracy of both networks, demonstrating the Mean Teacher model’s superiority over the $\Pi$ model with 4000-label datasets.
  Furthermore, as shown in Figure 16, the Mean Teacher model excels in terms of loss, even with suboptimal hyperparameter tuning, highlighting the efficacy of the network design with the same core network and identical hyperparameters.
• Mean Teacher with DenseNet and Shake-Shake26: We deemed it essential to test the state-of-the-art SSML Mean Teacher model with a core network other than Shake-Shake26, hence the choice of DenseNet-121. Figure 17 shows that the training loss for both student networks is predictably lower, as the teacher network guides them using the EMA formula. These outcomes suggest that the teacher’s parameters are optimized for the network’s best overall performance. Moreover, the loss from the teacher represents the most optimal value attainable.
  Additionally, the Shake-Shake26 network significantly outperforms DenseNet-121. As seen in Figure 18, this superiority is also reflected in accuracy performance. The primary reason is Shake-Shake26’s broader and deeper network compared to DenseNet-121. However, it requires more computational time for training and hyperparameter tuning. System designers must consider this computational overhead, particularly when prioritizing performance over flexibility across datasets.
  Figure 19 provides a few intriguing observations about the relationship between the teacher and the student models. First, with data with 4000 labels, the Shake-Shake26 network training process experiences a huge surge in the loss in the student nework. As a result, the teacher’s network progress is also impacted. Secondly, the accuracy performance of a particular neural network is different for different datasets with various percentages of the labeled dataset. Therefore, it implies that the neural network design must be modified to incorporate dataset variations. Moreover, the hyperparameter tuning in the EMA can render better assumptions that would help in certain scenarios.

Summary:

The incorporation of dual DNNs within a network classifier has shown to confer improved accuracy and reduced testing loss, a likely consequence of the combined classification prowess during forward propagation. This dual-network setup also injects additional variability into the calculation of the consistency cost, which in turn influences the total loss, with unlabeled data contributing to this effect. A comparative assessment of semi-supervised machine learning (SSML) models indicates that the Mean Teacher model surpasses the $\Pi$ model in terms of accuracy and achieves a lower test loss.

When examining the performance across different SSML algorithms using Shake-Shake26, this model consistently presented a reduced loss when pitted against the DenseNet and WideResNet models, which may be attributable to its network width.

6.2. PF-SSCP with Different Network Classifiers

Understanding how PF-SSCP compares to CF-SSCP is crucial, but it is also vital to discern how the network classifier impactsPF-SSCP. We conducted tests using different network classifiers within both the MixMatch and ReMixMatch frameworks. The subsections below detail our analysis.

Results of MixMatch and ReMixMatch Analysis

• MixMatch Compared to ReMixMatch with WideResNet as a Core Network:
  Given PF-SSCP’s emphasis on performance with the increased use of unlabeled data, we compared the outcomes from both frameworks using the same WideResNet-28-2, which has a depth of 28, a width of 2, and incorporates batch normalization [34]. As depicted in Figure 20, ReMixMatch slightly outperforms MixMatch with 4000 labels. This discrepancy widens with fewer labeled data.
  With 1000 labels, as shown in Figure 21, the performance gap becomes more pronounced. An intriguing observation is the convergence of the two semi-supervised machine learning models (SSMLs) within a limited number of epochs. ReMixMatch achieves higher initial accuracy, indicating its strength with fewer labeled data. However, overall performance differences are marginal, as MixMatch eventually catches up by the end of training.
• MixMatch and ReMixMatch with Shake-Shake26 as a core network:
  Employing different core networks allows us to visualize the performance contributions of the SSMLs, independent of the underlying networks. By switching to Shake-Shake26, we sought to discern any performance variations. A close examination with 1000 labels, as seen in Figure 22, reveals an insignificant difference compared to WideResNet-28-2. Shake-Shake26 converges more rapidly, with marginally better accuracy, but this is not as pronounced when these Deep Neural Networks (DNNs) are used independently.
  In Figure 23, with 4000 labels, ReMixMatch paired with Shake-Shake26 shows the highest accuracy. Notably, ReMixMatch with WideResNet and MixMatch with Shake-Shake26 are closely matched. The accuracy of ReMixMatch with WideResNet is slightly higher, but only by a narrow margin. Another key observation is the divergence in convergence patterns between MixMatch with Shake-Shake26 and MixMatch with WideResNet, as confirmed through five consecutive tests to ensure the finding’s accuracy.

Summary:

A comparative review of the experimental data reveals that MixMatch and ReMixMatch algorithms yield higher accuracy improvements over the Mean Teacher, $\Pi$ model, and Temporal Ensembling methods, underscoring the significance of preprocessing in model performance. When juxtaposing MixMatch with ReMixMatch, the latter exhibits a marginal lead in accuracy, indicating its slight edge within the preprocessing-enhanced learning approaches.

6.3. Training Time

Since the training time of a neural network in a production environment is critical, choosing the correct DNN for the SSML is a major decision in some cases where data change.

Table 3. Pearson correlation coefficients.

Network Classifier	SSML Algorithm	Labeled Data/ Unlabeled Data	Training Time (min)	PCC	Average PCC
Shake-Shake26	Temporal E.	49,000/1000	1020	$PC{C}_{shake}=0.7227$ p-value$=0.002003$ $z_{shake}^{\prime }=0.929$	$PC{C}_{average}=0.701$ p-value$=0.003597$ $z_{avg}^{\prime }=0.87$
		46,000/4000	824
		0/50,000	120
	Mean Teacher	49,000/1000	1161
		46,000/4000	1094
		0/50,000	158
	$\Pi$ model	49,000/1000	1351
		46,000/4000	1272
		0/50,000	168
	MixMatch	49,000/1000	1211
		46,000/4000	1103
		0/50,000	163
	ReMixMatch	49,000/1000	1239
		46,000/4000	1142
		0/50,000	149
DenseNet-121	Temporal E.	49,000/1000	323	${\mathit{PCC}}_{\mathrm{densenet}}=\mathbf{0}.\mathbf{7585}$ p-value$=\mathbf{0}.\mathbf{001046}$ ${\mathit{z}}_{\mathit{densenet}}^{\prime }=\mathbf{0}.\mathbf{993}$
		46,000/4000	276
		0/50,000	110
	Mean Teacher	49,000/1000	392
		46,000/4000	337
		0/50,000	124
	$\Pi$ model	49,000/1000	443
		46,000/4000	407
		0/50,000	168
	MixMatch	49,000/1000	404
		46,000/4000	353
		0/50,000	131
	ReMixMatch	49,000/1000	411
		46,000/4000	367
		0/50,000	139
WideResNet	Temporal E.	49,000/1000	387	$PC{C}_{wide}=0.5968$ p-value$=0.01904$ $z_{wide}^{\prime }=0.688$
		46,000/4000	324
		0/50,000	132
	Mean Teacher	49,000/1000	537
		46,000/4000	485
		0/50,000	158
	$\Pi$ model	49,000/1000	553
		46,000/4000	512
		0/50,000	179
	MixMatch	49,000/1000	551
		46000/4000	512
		0/50,000	149
	ReMixMatch	49,000/1000	567
		46,000/4000	517
		0/50,000	151

presents the training time results for every combination of all DNNs by executing all the SSMLs on the GPU—i.e., an Nvidia 2080 Ti GTX. The deeper and wider DNN is Shake-Shake26, and for this reason, the results demonstrate the highest training time in this case. On the other hand, the lowest training time was observed for DenseNet-121. This is due to the interconnection among the layers [35], making the total number of parameters lower as compared to the WideResNet model.

Furthermore, in case of the $\Pi$ model, the training time is higher due to the model’s ability to perform backward propagation for both the student and the teacher models. Expectedly, the lowest training time is still that of Temporal Ensembling, as it includes only one DNN in the classifier, which reduces the total number of parameters and the SSML’s complexity.

Adding the PF-SSCPs, MixMatch and ReMixMatch require more training time as compared to the CF-SSCPs, excluding the $\Pi$ model. This is the due to the complexity of the preprocessing step introduced and the pseudo-label calculation to generate the labeled and unlabeled training sets. Comparing MixMatch and ReMixMatch, it is clear that ReMixMatch requires more training time due to its more complex calculation of the pseudo-labels, as shown in Figure 4.

In order to see if the correlation would still withhold, Table 3 presents the Pearson correlation coefficients (PCC) [36] and the training time for all network classifiers independent of the SSCP with different SSML algorithms.

As shown previously, the training times for the Shake-Shake26 model as the DNN with a 49,000/1000 ratio of unlabeled/labeled data are shown as 1020, 1161, 1351, 1211, and 1239 min using the Temporal Ensembling model, Mean Teacher model, $\Pi$ model, MixMatch, and ReMixMatch, respectively. On the other hand, the training times are 120, 158, 168, 163 and 149 min when there are no labels. In Table 3, we have similar results for the other network classifiers. The Pearson correlation coefficients (PCC) for each network classifier are used to indicate a positive correlation between the training time and the labeled/unlabeled data ratio.

Looking at $PC{C}_{shake}$, we can observe a correlation of 0.7227 with a p-value of 0.002003, compared to utilizing only the CF-SSCP algorithm with a p-value of 0.0278 and $PC{C}_{shake}$ of 0.7227. Moreover, we can observe that $0.73>0.7227>0.7$ and $0.002003<0.0278<0.05$, which indicates a higher correlation. We can find similar results for the DenseNet model with a $PC{C}_{DenseNet}$ value of 0.7585. However, for WideResNet, we see a weaker correlation, although it is noticeably positive with a $PC{C}_{WideResNet}$ of 0.5968. To evaluate the average correlation coefficient among all three network classifiers, we used Fisher’s Z to transform each correlation coefficient. Subsequently, we calculated the mean $z^{\prime }$ value in each case. This means $z_{avg}^{\prime }$ value was then transformed to the correlation coefficient again. After performing these computations, the $PC{C}_{avg}$ was found to be 0.701, which is greater than 0.692, as stated with CF-SSCP, which indicates a higher correlation.

7. Discussion

As observed in the experiments, SSML based on PF-SSCP demonstrates higher accuracy and lower loss compared to that based on CF-SSCP, directly informing our conclusion regarding RQ1. This improved performance can be attributed to the quantity of data inputted into the DNNs, which plays a pivotal role in an SSML’s effectiveness. This is due to the utilization of parameter updates in each design. Additionally, for both PF-SSCP and CF-SSCP network classifiers, we observed that fewer skip connections enhanced accuracy and diminished loss. This diversification of parameter updates facilitated more effective forward propagation, influencing the consistency cost.

Another insight gleaned from these experiments pertains to the correlation between training time and the ratio of labeled to unlabeled data, as addressed in RQ2. The presence of more labeled data necessitated increased computational time to calculate the loss, primarily due to the classification cost. Alterations in the classification cost significantly affected the loss, prompting substantial parameter updates during backward propagation, which, in turn, impacted subsequent forward propagation.

8. Conclusions

In this paper, we delved into various facets of SSML through the lenses of post-hoc XAI and global scale analysis. We dissected SSML into two distinct SSCP components, highlighting the disparities in learning precision, training time, and design when integrated with cutting-edge DNNs as network classifiers. Through detailed graphs, we elucidated their operational mechanics, pinpointing the factors that contribute most significantly to their functionality.

Moreover, our research, grounded in 45 rigorous experiments, substantiated the impact of preprocessing techniques introduced within the PF-SSCP method. Our analyses revealed that the choice of a DNN as a network classifier, regardless of the SSML algorithm employed, profoundly influences the magnitude of parameter updates, both in terms of frequency and size, particularly when more skip connections are present. Additionally, our findings indicated that the preprocessing stages within SSML exert a more pronounced effect than the selection of the network classifier itself. This insight offers a strategic direction for ML developers and engineers, encouraging the adoption of innovative frameworks within SSMLs and DNNs to fully harness the potential of domain-specific data.

Building on prior discoveries, we further reinforced the observed correlation between training time and the ratio of labeled to unlabeled data, providing robust statistical evidence to support this relationship. In future endeavors, we aim to meticulously analyze and deconstruct groundbreaking SSML frameworks in conjunction with DNNs. Our goal is to authenticate the efficacy of the contributing factors identified in this study by exploring a broader spectrum of cases.