A non-negative feedback self-distillation method for salient object detection

Salient object detection

Traditional SOD methods mainly use color (Cheng et al., 2014b), boundary prior (Zhu et al., 2014), and sparsity (Li, Sun & Yu, 2015) to obtain the object salient map. These methods can obtain more detail information, but they are difficult to obtain high-quality semantic information. The recently developed deep learning methods can obtain both the detail and semantic information, so more and more scholars (Wei et al., 2020; Chen et al., 2020; Zhao et al., 2020; Wei, Wang & Huang, 2020; Wu, Su & Huang, 2019; Mao et al., 2021) use deep learning methods to solve SOD problem. Wei et al. (2020) divided the feature map into two parts: body map and detail map, and refined the two parts respectively. Chen et al. (2020) adopted feature interweaved aggregation, self-refined, head attention and global context flow modules to construct a global context-aware progressive aggregation network. Zhao et al. (2020) designed a gated dual branch structure. Collaborations between features from different layers were established to improve the distinguishability of the entire network. Wei, Wang & Huang (2020) mixed features from different layers by designing cross feature modules and cascaded feedback decoders, to generate better salient maps. Wu, Su & Huang (2019) discarded the features from shallow layers to improve the computing efficiency, and refined the features from deep layers to improve their representation ability. Chen et al. (2020) introduced initial prediction, side-output residential learning and top-down reverse attention to solve the complex architecture problem. Mao et al. (2021) used swin transformer structure to mix multi-layer features, and used attention mechanism to strengthen feature representation ability. However, these methods usually have large network structures and are difficult to be directly applied to the reality.

Recently, Zhang et al. (2019b) applied KD to SOD, they reduced the channels amount to construct the student network, and adopted multi-scale to transfer knowledge from the teacher to the student. Besides Piao et al. (2020) introduced cross-modal distillation on RGB-D based SOD, they distilled the depth information through an adaptive distiller. However, less attention is paid to self-distillation to solve SOD problem.

Self-distillation

Ordinary knowledge distillation requires an additional teacher network to guide the student network learning. Self-distillation methods do not need an additional teacher network, and the student network learn the knowledge from itself. Self-distillation methods mainly construct auxiliary branches to transfer the knowledge to the student network. Ji et al. (2021) used the auxiliary teacher network to refine the soft label and feature map knowledge, which can better preserve the local information. Hou et al. (2019) let the shallow feature learn the deep feature expression, so as to strengthen the overall feature expression ability. Zhang et al. (2019a) divided the original network into several shallow networks according to the characteristics of the network structure, and distilled the separated shallow networks respectively. Li & Chen (2020) used consistency strategy to match the knowledge that is extracted from auxiliary branches.

Li et al. (2020) proved that self-distillation is a special label smoothing regularization method. Yun et al. (2020) used the predicted distribution of training samples as distillation knowledge. Therefore, Li et al. (2020) and Yun et al. (2020) considered that self-distillation is a special regularization method.

At the same time, Xu & Liu (2019) and Lee, Hwang & Shin (2020) considered that self-distillation is a special data augmentation method. Xu & Liu (2019) used the distorted versions of training samples as distillation knowledge. Lee, Hwang & Shin (2020) used the transformations of training samples as distillation knowledge.

The regularization method does not require complex branch structures and matching strategies, and can further simplify the scale of network parameters. We use pixel-wise regularized distribution to construct a self-distillation framework.

Virtual teacher self-distillation architecture

Li et al. (2020) used the regularized category probability as virtual knowledge to construct a teacher model, and manually set the output probability of the teacher network. Based on this, we used the regularized probability distribution to construct a pixel-wise virtual teacher model. Unlike Li et al. (2020), who addressed multi-classification problems, we have extended their approach to tackle pixel-wise segmentation problem. The self-distillation structure is shown in Fig. 1. The output probability of the teacher network is as follows: (1)p t a i = μ , i f a i = y i 1 − μ K − 1 , i f a i ≠ y i where K is the total number of categories, K is 2 in SOD; y_i is the correct label; and a_i is the prediction label; µis the predict probability of correct pixel classification. Usually, μ ≥ 0.9 is set (Li et al., 2020), to ensure that the probability of correct pixel classification is far greater than wrong classification; we set μ = 0.99. When the pixel is the labeled foreground, the output probability of the virtual teacher is 0.99; when the pixel is the labeled background, the output probability of the virtual teacher is 0.01. If the value of µis small, the predict probability of the student will be greater than the teacher in easy and well classified pixels. In these pixels, the teacher cannot transfer knowledge to the student, and will bring negative effect to the network training. Therefore, we set µas 0.99, and Li et al. (2020) proved it in experiments.

We use the backbone network of F3Net (Wei, Wang & Huang, 2020) as the student network (the right box in Fig. 1), and build a self-distillation learning framework on this basis. The virtual teacher provides correct knowledge to the student network, and guild the student network to optimize. Different from hard label learning which hopes the output of the teacher and student are the same, this self-distillation method hopes that the output distribution of the student fits the teacher output distribution. Virtual teacher self-distillation method provides more distribution information while making the student results the same as the teacher.

For a training set with N samples, D = X j , Y j | 1 ≤ j ≤ N, where X_j ∈ R^H∗W∗3 is the j-th sample, H and W are the height and width of the sample, Y_j is the corresponding ground truth. W M = W m | m = 1 , … , L represents the learnable weight matrix of a L-layer neural network. The training goal of the neural network is to learn a mapping function f W m ; X : X → Y. The most common training method is Empirical Risk Minimization (Wang, 2021). The neural network parameter W_m can be adjusted by optimizing the following functions. (2)Arg min W m L m t W m ; D , where L_mt is the total loss of all training samples. In the self-distillation framework for SOD, the loss function is determined by pixel-wise CE and KL loss. (3)L m t = L K L p s , p t + L C E W m ; D , (4)L K L p s , p t = 1 N ∗ W ∗ H ∑ j = 1 N ∑ i = 1 W ∗ H p i j s log p i j s p i j t , (5)L C E W m ; D = − 1 N ∗ W ∗ H ∑ j = 1 N ∑ i = 1 W ∗ H y i j log p i j s + 1 − y i j log 1 − p i j s , where p i j s is the prediction probability of the student network for i-th pixel in j-th sample; p i j t is the prediction probability of the corresponding teacher network; y_ij is the corresponding annotated label, which is 1 when the pixel belongs to the foreground and is 0 when the pixel belongs to the background.

It is found that this virtual teacher self-distillation architecture can improve the model performance for classification tasks, but achieve little improvement for SOD. Especially on easily classified datasets with significant differences between foreground and background, the model performance can hardly be improved. To apply the virtual teacher self-distillation method to SOD, we analyze KL loss and propose a new loss to replace KL loss.

Distillation loss analysis in classification task

In classification task, self-distillation structure usually uses Cross Entropy loss ( L_CE) and Kullback–Leibler divergence loss ( L_KL) to guide the student network training (Kim et al., 2021). The sample loss L is calculated as follows: (6)L = L K L p s p t + L C E p s , Y . In binary classification task, Cross Entropy (CE) is Binary Cross Entropy (Li et al., 2019). Hossain, Betts & Paplinski (2021) proved that when the pixel belongs to the object, CE loss is positive and the loss derivative is negative; when the pixel belongs to the background, CE loss is positive, and the loss derivative is positive. The formulas of CE loss and loss derivative are shown in Table 1, the loss curves are shown in Fig. 2, and the loss derivative curves are shown in Fig. 3. The optimization direction of a good distillation loss should be consistent with CE. If the optimization direction is not consistent, the negative feedback will be generated. The negative feedback affects the network optimization, and leads to the poor performance.

KL divergence L_KL is calculated by Eq. (7) (Li et al., 2020). As the temperature parameter does not affect the optimization direction of the loss and loss derivative, the influence of temperature parameter is not considered. (7)L K L p s p t = 1 W ∗ H ∑ i = 1 W × H p i s → log p i s → p i t → , where p i s → is the output of the student network, which denotes a 1 ×k dimensional array. K is the number of categories, it is 2 in binary classification task. p i t → is the output of the corresponding teacher network. (8)p i s → = p 1 i s , p 2 i s , … , p k i s , where p 1 i s , p 2 i s , … , p k i s are the output probability predicted by the softmax function, and their summation is 1. In binary classification task, the output of the student network can be expressed as follows: (9)p i s → = p i s , 1 − p i s , where p i s is the probability that the student network predicts the pixel i as the object. The KL loss of the pixel i is calculated as follows: (10)L K L i p i s → p i t → = 1 2 p i s log p i s p i t + 1 − p i s log 1 − p i s 1 − p i t , where p i t is the probability that the teacher network predicts the pixel i as the object, and the value of p i t is (0,1).

The derivation of L K L i is as follows: (11)d L K L i d p i s = 1 2 log p i s p i t + log 1 − p i t 1 − p i s . In self-distillation framework, most works construct auxiliary teacher branches to generate refined knowledge, or adopt deep-level knowledge to guild the shallow-level training. The purposes of these works are to construct the teacher which performance is better than the student network (Hinton, Vinyals & Dean, 2015). When the pixel belongs to the object, the output probability of the teacher network is greater than the student network, that is p i t ≥ p i s. So d L K L i / d p i s ≤ 0, L K L i is a monotone decreasing function in the range of values. p i t ≥ p i s, the maximum value of p i s is p i t . L K L i p i s = p i t = 0, the minimum value of L K L i is 0. So, L K L i is greater than 0. Therefore, when the pixel belongs to the object, the distillation loss calculated by KL divergence is greater than 0 and the loss derivative is less than 0. In the object, the optimization directions of the distillation loss and loss derivative are consistent with CE.

When the pixel belongs to the background, the output probability of the teacher network is less than the student network, that is p i t ≤ p i s. So d L K L i / d p i s ≥ 0, L K L i is a monotone increasing function in the range of values. p i t ≤ p i s, the minimum value of p i s is p i t . L K L i p i s = p i t = 0, the minimum value of L K L i is 0. So, L K L i is greater than 0. Therefore, when the pixel belongs to the background, the distillation loss calculated by KL divergence is greater than 0 and the loss derivative is greater than 0. In the background, the optimization directions of the distillation loss and loss derivative are consistent with CE.

For a more intuitive presentation, we assume that the teacher network output probability p i t is 0.99 in the object pixel and 0.01 in the background pixel. At this time, the distillation loss and loss derivative formulas are shown in Table 2, the loss curves are shown in Fig. 2, and the loss derivative curves are shown in Fig. 3.

From Figs. 2 and 3, it can be seen that the signs of distillation loss and loss derivative value are consistent with CE. As the optimization direction of the distillation loss which is calculated by KL divergence is consistent with CE, distillation loss can better guide the student network training in binary classification task.

Distillation loss analysis in salient object detection

Similar to the classification task, we also use CE loss (L_CE) and distillation loss (L_KD) to guide the student network training in SOD. Then the sample loss L is calculated as follows: (12)L = L K D p s , p t + L C E p s , Y . In SOD, CE is also the binary CE (Wei, Wang & Huang, 2020). That is, when the pixel belongs to the foreground, CE loss is positive and the loss derivative is negative. When the pixel belongs to the background, CE loss is positive, and the loss derivative is positive.

When KL formula is used to calculate the distillation loss in SOD, the distillation loss is calculated as follows: (13)L K D p s , p t = L K L p s p t = 1 W ∗ H ∑ i = 1 W × H y i p i s log p i s p i t + 1 − y i p i s log p i s p i t , where y_i = 0 means the pixel belongs to the background; y_i = 1 means the pixel belongs to the foreground; p i s and p i t are the outputs of the student and teacher networks, which are 1-dimensional values. The KL loss of the pixel i is calculated as follows: (14)L K L i p s p t = p i s log p i s p i t . The derivation of L K L i is as follows: (15)d L K L i d p i s = log p i s p i t + 1 . In self-distillation framework, most works construct auxiliary teacher branches to generate refined knowledge, or adopt deep-level knowledge to guild the shallow-level training. The purposes of these works are to construct the teacher which performance is better than the student network (Hinton, Vinyals & Dean, 2015). When the pixel belongs to the foreground, the output probability of the teacher network is greater than the student network, p i t ≥ p i s . L K L i p s p t is less than 0. At this time, the sign of distillation loss is inconsistent with CE. When 0 < p i s / p i t ≤ 1 / e, that is d L K L i / d p i s ≤ 0. At this time, the sign of loss derivative is consistent with CE. When 1 / e < p i s / p i t ≤ 1, that is d L K L i / d p i s ≥ 0. At this time, the sign of loss derivative is inconsistent with CE. Therefore, in the foreground, the optimization direction of distillation loss will be inconsistent with CE, resulting in negative feedback, which will affect the student network performance.

When the pixel belongs to the background, the output probability of the teacher network is less than the student network, p i t ≤ p i s . L K L i p s p t is greater than 0; d L K L i / d p i s is greater than 1. At this time, the optimization directions of distillation loss and loss derivative are consistent with CE. For a more intuitive presentation, we assume that the teacher network output probability p i t is 0.99 in the foreground pixel and 0.01 in the background pixel. The distillation loss and loss derivative formulas are shown in Table 3, the loss curves are shown in Fig. 2, and the loss derivative curves are shown in Fig. 3.

Combining Figs. 2 and 3 and the above analysis, it can be seen that when KL formula is used to calculate the distillation loss in SOD, the optimization direction of the distillation loss and loss derivative are inconsistent with CE in the foreground. The performance improvement of the student network is limited. Therefore, it is defective to directly use KL formula to calculate the distillation loss in SOD.

Non-negative feedback distillation loss (NKL)

In order to transfer the knowledge, the optimization direction of distillation loss should be consistent with CE. In SOD, when the pixel belongs to the foreground, the loss is greater than 0, and the loss derivative is less than 0; when the pixel belongs to the background, the loss is greater than 0, and the loss derivative is greater than 0.

Inspired by KL, CE and Focal loss (Lin et al., 2017), we propose a non-negative feedback distillation loss, which uses different formulas to respectively calculate foreground and background distillation loss. The loss is calculated as follows: (16)L N K L p s , p t = 1 W ∗ H ∑ i = 1 W × H y i 1 − p i s α log 1 − p i s 1 − p i t + 1 − y i p i s α log p i s p i t . where y_i = 0 means the pixel belongs to the background; y_i = 1 means the pixel belongs to the foreground; α is a hyperparameter that is greater than 0 and less than 1, which is determined by experiments and is selected as 0.3 here.

When the pixel belongs to the foreground, the distillation loss and loss derivative of the pixel i are calculated as follows: (17)L N K L i + p s , p t = 1 − p i s α log 1 − p i s 1 − p i t , (18)d L K L i + d p i s = α 1 − p i s α − 1 log 1 − p i t 1 − p i s − 1 − p i s α − 1 . At this time, 1 ≥ p i t ≥ p i s ≥ 0. Therefore, L N K L i + p s , p t is greater than 0, d L K L i + / d p i s is less than 0. Therefore, when the pixel belongs to the foreground, our distillation loss is greater than 0 and the loss derivative is less than 0. In the foreground, the optimization directions of our distillation loss and loss derivative are consistent with CE.

When the pixel belongs to the background, the distillation loss and loss derivative of the pixel i are calculated as follows: (19)l L N K L i − p s , p t = p i s α log p i s p i t , (20)d L K L i − d p i s = α p i s α − 1 log p i s p i t + p i s α − 1 . At this time, 0 ≤ p i t ≤ p i s ≤ 1. Therefore, L N K L i − p s , p t is greater than 0, d L K L i − / d p i s is greater than 0. Therefore, when the pixel belongs to the background, our distillation loss is greater than 0 and the loss derivative is greater than 0. In the background, the optimization directions of our distillation loss and loss derivative are consistent with CE. For a more intuitive presentation, we assume that the teacher network output probability p i t is 0.99 in the foreground pixel and 0.01 in the background pixel. The distillation loss and loss derivative formulas are shown in Table 4, the loss curves are shown in Fig. 2, and the loss derivative curves are shown in Fig. 3.

As the optimization direction of our non-feedback distillation loss is consistent with CE in the foreground and background, it can better guide the student network training in SOD.

So, in virtual teacher self-distillation architecture, we replace Eq. (4) as Eq. (21). (21)L N K L p s , p t = 1 N ∗ W ∗ H ∑ j = 1 N ∑ i = 1 W ∗ H y i j 1 − p i j s α log 1 − p i j s 1 − p i j t + 1 − y i j p i j s α log p i j s p i j t . In order to verify the universality of our method, we also apply our non-negative feedback distillation loss to other self-distillation frameworks. Through experiments in ‘Comparison with recent self-distillation methods’ we find that our non-negative feedback loss function can work in other self-distillation methods and work better in our virtual teacher method. The main reason is that the teacher network performance may be worse than the student in easy and well classified pixels in other self-distillation methods. In non-negative feedback loss function, the worse performance teacher cannot guild the student training, which leads to the limited improvement to the student network. While in the virtual teacher method, the teacher performance is always better than the student.

Experimental configurations

Comparison with different one-dimensional distance metric methods

In SOD, the output is the category probability which is one-dimension, the aim of self-distillation is that the student network produces the same distribution with the teacher. Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Mean Squared Error (MSE), Cosine Similarity (CS) are the common one-dimensional distance metric functions and are widely used as the loss functions. Therefore, we use these functions and KL formula (KL) as distillation loss functions, and compare them with our method (NKL). In the experiment, virtual teacher self-distillation architecture remained unchanged and only changed the distillation loss function. The experiment results are shown in Table 6.

The following conclusions can be drawn from Table 6. (1) Not all one-dimensional distance metric methods can be used as distillation loss function. From the mean performance over five datasets, when RMSE is used as the distillation loss, the network performance after distillation is poorer than before distillation. From the evaluating indicator MAE, when KL formula is used as distillation loss, the network performance after distillation is poorer than before distillation. (2) Our method (NKL) can transfer the knowledge well. Our method can achieve the best detection results on all five datasets. Especially, from the mean E, our method improves 2.2% compare with the second-best method (KL); from the mean MAE, our method reduces 1.6% compare with KL. These prove that our method is effective.

Comparison with recent self-distillation methods

First, we use the backbone network of F3Net as the baseline and take this as the student network. Then, FR (Ji et al., 2021), SA (Hou et al., 2019), BYOT (Zhang et al., 2019a) and DHM (Li & Chen, 2020) self-distillation methods are introduced into the baseline. Finally, KL formula and our non-negative feedback loss function (NKL) are respectively used as distillation loss to train the network. The experiment results are shown in Table 7.

SA directly uses pixel-wise attention features in the backbone network as distillation knowledge. FR uses Bi-directional Feature Pyramid Network (BiFPN) to generate pixel-wise knowledge. Therefore, SA and FR can be directly applied to SOD. BYOT and DHM cannot directly generate pixel-wise knowledge. Therefore, we modify them to ensure that they can be applied to the self-distillation framework for SOD. We mainly make the following two modifications. (1) We adjust the step size of the first convolution layer in the bottleneck module from 2 to 1. This operation is to maintain the spatial size of the feature in the bottleneck module of the auxiliary branch. (2) We change the full connected layer of the auxiliary branch to the convolution layer. This operation is to generate pixel-wise knowledge that can be transferred to the backbone network.

From Table 7, we draw the following conclusions. (1) Self-distillation methods are also suitable for SOD. The results in the table show that self-distillation methods can improve the network performance. Compared with the baseline, our method improves the average F by nearly 2% and the average E by nearly 3.8%. (2) Our virtual teacher model is better than other self-distillation methods. In the five datasets, our virtual teacher model can achieve the best detection results. Especial, from the mean F, our method improves nearly 2% compare with the second-best method (BYOT); from the mean E, our method improves nearly 3% compare with the second-best method (BYOT). (3) Our non-negative feedback loss function (NKL) achieves better results than KL formula in different self-distillation methods in SOD. In DHM, NKL can achieve better detection results in five datasets. Among other methods, NKL can also achieve better results in at least three datasets. And KL is mainly better than our method in COD. As the foreground and background are similar in camouflaged images, the prediction result of the teacher network may be worse than that of the student network in other self-distillation methods. In NKL, the worse performance teacher cannot guild the student training, which limits the improvement of the network performance. (4) NKL can work better in our virtual teacher method than other self-distillation methods. From the mean E, NKL improves nearly 2% compare with KL in virtual teacher, but improves nearly 0.3% in other self-distillation methods. The main reason is that the teacher network performance may be worse than the student in easy and well classified pixels in other self-distillation methods. In NKL, the worse performance teacher cannot guild the student training, which leads to the limited improvement to the student network.

Comparison with recent SOD methods

We compared with five SOD methods, namely LDF (Wei et al., 2020), GCP (Chen et al., 2020), RAS (Chen et al., 2020), GateNet (Zhao et al., 2020), CPD (Wu, Su & Huang, 2019). All methods are used their default learning rate, momentum, weight decay and maximum epochs.

Table 8 quantitatively shows the detection results of different methods. It can be seen that our method can achieve good detection results on five datasets, and can achieve the best detection results on THUR and HKU-IS. From the mean performance over five datasets, our method also achieves the best detection results. From the mean E, our method improves nearly 3% compare with CPD.

Figure 4 shows the precision–recall curves of different methods. It can be seen that our curves are higher than other methods in COD, THUR and HKU-IS, which prove that our method can achieve good performance.

Table 9 shows the detection efficiency of different methods. We compare the model efficiency from the model parameter size and detection speed. It can be seen that our method has the smallest parameter scale and the fastest detection speed, when our model performance is similar to other methods. From the size of params, our method reduces near 100M compare with GateNet.