Distilling Knowledge in Machine Translation of Agglutinative Languages with Backward and Morphological Decoders

In the realm of Statistical Machine Translation (SMT), Koehn and Hoang [19] pioneered the concept of factored translation models, integrating diverse morphological features into the translation process. They accomplished this by augmenting word representations with additional morphological and syntactic features. Unlike SMT, Neural Machine Translation (NMT) models use a fixed vocabulary and hence use a vocabulary reduction method like BPE [32]. However, previous works like Banerjee and Bhattacharyya [4] shows that using a linguistically motivated morfessor improves translation in some languages, while Weller-Di Marco and Fraser [42] use a rule-based morphological analyzer for translation tasks in English-German translation. Subsequently in Neural Machine Translation, a stream of research has emerged aiming to enhance word vector representations by incorporating valuable linguistic features into either the source encoder or the target decoder. For instance, Sennrich and Haddow [31] broadened the embedding layer of an NMT encoder by incorporating a blend of morphological and syntactic features, encompassing word lemma, morphological attributes, POS, and dependency labels. Their model utilized concatenated feature embedding vectors as input word embeddings while keeping the rest of the NMT model unaltered. Conversely, Bandyopadhyay [3], Song et al. [39], and Tamchyna et al. [40] factored words into morphological (lemma) and syntactic (factors) features at the output decoder of NMT. Their model, augmented with a heuristic morphological synthesizer, generated unseen word forms based on predicted lemma and factors. This method aimed to address challenges related to large vocabulary and out-of-vocabulary (OOV) instances during translation. Regrettably, this approach did not yield significant improvements in experimental outcomes. Incorporating these features on the target side of the translation system is challenging since it requires incrementally tagging and parsing the hypotheses at test time, which requires the whole sentence, not available in test time. Nzeyimana [26] tried to incorporate linguistic tags through multi-tasking multi-label training. Our paper advocates for integrating linguistic knowledge in the target side words by distilling knowledge from a morphological decoder.

Another line of research explores the integration of external knowledge through multi-source or multi-task learning. This involves considering additional sources as valuable, distinct information that augments the learning process of a translation model. The idea behind these approaches is to integrate the linguistic features into the model architecture, such as in a multi-encoder [22], modified attention [5, 49], or multi-task learning [8].

To incorporate future information into neural machine translation (NMT), various approaches have been proposed. One approach is to use reinforcement learning, such as the REINFORCE algorithm [34, 43, 45, 46] or the actor-critic algorithm [1, 21].

A separate set of techniques incorporates future information into the inference process by employing additional decoding passes or supplementary components during testing. For instance, Xia et al. [44] and Zhang et al. [50] advocated a two-pass decoding algorithm, generating an initial draft translation followed by a refined final translation based on the draft. In a similar vein, Zhang et al. [48] and Zhou et al. [54] maintained both forward and backward decoders, decoding simultaneously and interacting when making predictions. From a distinct perspective, some researchers focus on integrating future information, such as Feng et al. [9] and Zhang et al. [47] which propose leveraging future source information to guide machine translation with knowledge distillation, mitigating source incompleteness. Another approach is to model past and future information for the source to help the decoder focus on untranslated source information, such as in Zheng et al. [53] and Zheng et al. [52]. However, our method differs from previous approaches in that it selectively distills knowledge from the additional decoder only when the knowledge is reliable and takes the reliability of future information into account.

Morphological complexity varies across languages, and it is important to identify this complexity in order to apply our method. Instead of relying on expert linguistic descriptions, we adopt corpus-based metrics such as Types, Type-Token Ratio (TTR), and Moving-Average Type-Token Ratio (MATTR). These measures are effective in assessing and ranking languages based on morphological complexity, as suggested by Kettunen [18]. Table 2 summarizes these metrics.

Languages with rich morphology tend to exhibit high values for these metrics due to their diverse word forms, while languages with simpler morphology show lower values. By applying these measures to our selected languages, we obtain a quantitative basis for comparing their morphological characteristics, which will later inform the analysis of segmentation techniques in translation tasks.

Word-level models would predict “unknown” for out-of-vocabulary word tokens, making them unsuitable for morphologically rich languages with sparse vocabularies. Therefore, we train our translation models using two segmentation methods: BPE (Byte-Pair Encoding), a purely statistical method, and Morfessor, a linguistically motivated morphological analyzer.

Byte Pair Encoding. BPE [32] starts with character segmentation and merges characters into larger units based on their frequencies. This results in units that fall between characters and words, with the number of merge operations serving as a hyperparameter.

Morfessor. The default implementation [7] employs a unigram language model to identify morph-like structures. It selects segments in a top-down manner and includes a prior term for segment length, promoting the regularization of segments to resemble plausible morphemes. We use Morfessor Flatcat variant [14] for our implementation.

We now present the evaluation of our morphological measures and segmentation models. We experiment with the following languages: Tamil, Marathi, Manipuri, Japanese and Indonesian. These languages represent different families and degrees of morphological complexity, as reflected in our analysis.

Morphological Complexity Analysis. For this analysis, we use the multi-way parallel FLORES dataset [13]. Figure 1 compares the morphological measures across the five languages, alongside English. As seen in the figure, Tamil, Marathi, and Manipuri exhibit high morphological complexity, indicating their high agglutinative nature. In contrast, Japanese and Indonesian have relatively lower levels of agglutination.

Effects of Segmentation Methods on Machine Translation. To compare different segmentation models, we use the IWSLT 2017 dataset for Japanese and Indonesian. The English-Japanese dataset has 220,000 parallel sentences, and the English-Indonesian dataset has 76,000 parallel sentences. The experimental settings and dataset details for Tamil, Marathi, and Manipuri are provided in Section 5. Figure 2 shows the BLEU scores for both segmentation methods across these languages.

Our results demonstrate that Morfessor significantly improves translation quality for highly agglutinative languages. However, it does not offer an improvement for languages with a lower degree of agglutination, such as Japanese and Indonesian. In our further experiments, we will concentrate on the three highly agglutinative languages.

Our model is an extension of the dominant encoder-decoder model called the Transformer [41]. It employs a multi-decoder approach, in which separate decoders generate the target sentence in different ways.

Encoder and Forward Decoder. In the forward decoder, the target sentence \(Y={y_1,\ldots ,y_n}\) is generated by maximizing the conditional probability of each token \(y_t\) given the source sentence X and the preceding words \(y_{\lt t}\):The input sequence X is first fed into the encoder, which encodes it into m context vectors \(C = {c_1,c_2, \ldots , c_m}\) where m is the length of X. Specifically, this is achieved through the self-attention network (SAN), which produces the context vectors \(C = SAN(X)\).

The forward decoder generates the target sentence Y word-by-word based on the representation C from the encoder, using attention mechanisms to take into account the generated target fragment. Given a sequence of word embeddings \(\mathbf {E}(y_1),\ldots , \mathbf {E}(y_{t-1})\) in the generated target fragment at the t-th timestep, they are transformed into key matrix \(\mathbf {K}_{t-1}\) and value matrix \(\mathbf {V}_{t-1}\) . Another Self-Attention (SelfATTs) module is then used to learn the target representation \(s_t\):where \(\overrightarrow{h}_{t-1} \in \mathbb {R}^{d_{model}}\) is the previous context vector. \(s_t\) is then fed into another Attention (ATTc) module to compute the time-dependent context vector \(h_t\):where \(\mathbf {K}_e\) and \(\mathbf {V}_e\) are key and value matrices, respectively from encoder, that are transformed from the source representation C.

The probability distribution \(p(y_t|y_{\lt t}, X)\) of the generated target word \(y_t\) is computed using a multi-layer perceptron (MLP) layer:The target word \(y_t\) with the maximum probability is selected as the output of the decoder at the t-th timestep.

Backward Decoder. In the backward decoder, the target sentence is generated by taking into account the source sentence and the succeeding words:

To compute the probability of a target token \(y_t\), the decoder first computes the representation of the succeeding words using the self-attention mechanism. Given a sequence of word embeddings \(\mathbf {E}(y_{t+1}), ..., \mathbf {E}(y_{n})\) of the succeeding words at the t-th timestep, they are transformed into key matrix \(\mathbf {K}^{^{\prime }}_{t+1}\) and value matrix \(\mathbf {V}^{^{\prime }}_{t+1}\) . The target representation \(s^{^{\prime }}_t\) is computed using attention mechanism as:The context vector \(\overleftarrow{h_t}\) is then computed using attention mechanism as:The probability distribution \(p(y_t|y_{\gt t}, X)\) of the generated target word \(y_t\) is computed as:Figure 3 depicts the backward decoder (top) and forward decoder (bottom) with Reliable Knowledge Distillation.

Morphological Decoder. The morphological decoder generates the target sentence by taking into account the source sentence, the preceding words, and the morphological classes of the preceding tokens and the next token. The probability of a target token, \(y_t\), is computed using the classes of the preceding tokens and the next token \(class(y_{\le t})\), the preceding words, and the source sentence.In this morphological decoder, the word embeddings \(\mathbf {E}(y_1),\ldots , \mathbf {E}(y_{t-1})\) are combined with respective morphological class embeddings \(\mathbf {E}_c(class(y_1)),\ldots ,\mathbf {E}_c(class(y_{t-1}))\) by passing through a linear layer. The decoder is made aware of the morphological class of the next token to be generated in advance, allowing it to better choose the next token to generate based on this information. The morphological class of the next token, \(class(y_t)\), is passed to a class embedding, \(\mathbf {E}_c\), which generates its class embedding and combines it with the hidden state, \(h_t\) by passing through a linear layer. The hidden state \(h_t\) is computed in a manner similar to the forward decoder.

The probability distribution of the generated target word \(y_t\), \(p(y_t|class(y_{\le t}),y_\lt t, X)\) is computed as:Figure 4 depicts the morphological decoder (top) and forward decoder (bottom) with Reliable Knowledge Distillation. In our experiment, we take the morphological classes of stem or affix.

In this paper, we use knowledge distillation [16] to regularize the forward decoder in our machine translation model. The forward decoder has to match the probability distribution predicted by the backward decoder and morphological decoder. The knowledge distillation losses with forward decoder and morphological decoder are respectively expressed as:

To enhance the distillation effect, we dynamically weighted the knowledge distillation loss, which allows us to distill only reliable knowledge. We use the term “reliability” to indicate the level of confidence that we have in the prediction made by the ML model on a single example, as used in Kukar and Kononenko [20] and Nicora et al. [25]. Essentially, when the prediction error of the backward decoder or morphological decoder is high, the forward decoder will exclusively learn from the ground truth. The weighting factors \(\beta _t\) and \(\gamma _t\) are obtained from the inverse of cross-entropy between ground truth and predicted probability distribution as follows:where \(p^G(y_t)\) is the ground truth and \(\mathbb {K}\) is batchwise min-max normalization term.

In our approach, we train all three decoders using cross-entropy loss. The cross-entropy losses for the forward decoder, backward decoder, and morphological decoder are calculated as follows:where n is the number of tokens in each example.

We combine these cross-entropy losses with the additional knowledge distillation losses to obtain the final training losses as follows:We jointly train and update the parameters of all decoders. However, we are not updating the parameters of the backward decoder and morphological decoder using the knowledge distillation losses; instead, we only use the knowledge distillation losses to backpropagate and update the parameters of the forward decoder.

We evaluate our method on the following three datasets. WAT2021 English\(\rightarrow\)Tamil. (140K pairs): We evaluate using WAT2021 validation set and test set using PMIndia [15] and PIB Dataset [37]. English\(\rightarrow\)Manipuri. (120K pairs): We use a training set of 120K parallel sentences from PMIndia [15, 35] and PIB Dataset. The validation and test sets consist of approximately 1k sentence pairs each, sampled from the corpus. WAT2021 English\(\rightarrow\)Marathi. (132K pairs): We evaluate using WAT2021 validation set and test set using PMIndia and PIB Dataset.

To preprocess the data, we segment English words into subword units using Byte-Pair Encoding (BPE) [32] with 16,000 merge operations. Morphemes are splitted from the Manipuri, Tamil, and Marathi words using the Morfessor Flatcat tool [14]. The morphological class used in the Morphological Decoder consists of binary labels—“stem” vs. “affix”—for each sub-word token. We add a special marker “$$” before suffixes, and “+” between other morphemes of a word to ensure the reversibility of the morphological splitting. Certain words, like compound words and reduplicative words, contain multiple stems, which are indicated by the “+” symbol.

We conduct experiments on the following systems.

Transformer Baseline. A standard transformer model that is trained on the given dataset without incorporating any additional decoders.

Knowledge Distillation with Backward Decoder (KD with BD). A variation of our model that includes a backward decoder, which utilizes future information.

Reliable Knowledge Distillation with Backward Decoder (RKD with BD). Variation of KD with BD that distills only reliable knowledge.

Knowledge Distillation with Morphological Decoder (KD with MD). Another variation that includes a morphological decoder, which incorporates linguistic feature on the target side.

Reliable Knowledge Distillation with Morphological Decoder (RKD with MD). Variation of KD with MD but distills only reliable knowledge

ABDNMT. An NMT model based on the Transformer architecture, following the method proposed by Zhang et al. [50].

Twin Networks. The method proposed by Serdyuk et al. [33] and incorporates an L2 loss term.

SEER Forcing. An NMT model that utilizes the Seer forcing technique with knowledge distillation weight set to 0.5, following the method proposed by Feng et al. [9].

Factored Translation. the class information(stem or affix) of each token is obtained and both words and class are generated based on Bandyopadhyay [3] and Tamchyna et al. [40].

In this study, we conduct experiments using the Transformer [41] model, while implementing our models through adaptation from the open-source toolkit Fairseq-py [27]. There are three input and output layers with embedding dimension of 256, the inner feedforward layer dimension of 512, and the number of heads in the multi-head modules in both the encoder and decoder layers is 4. The training batches consist of sets of 4,096 source and target tokens. The models are trained and evaluated on two Tesla P100 GPUs. The test set is evaluated using a single model obtained by taking the best checkpoint, which is validated on the development set at each epoch. The BLEU metric [28] is used to evaluate the translation performance using the SacreBLEU [29].

Table 3 presents the results of our models and related works.

Morfessor models outperform BPE model. The baseline model, a standard Transformer trained without additional decoders, achieves moderate BLEU scores. Using BPE as the subword unit, the baseline obtains a BLEU score of 13.61 for English-to-Manipuri, 8.31 for English-to-Tamil, and 13.02 for English-to-Marathi. Using Morfessor for segmentation improves the performance to 14.32, 9.22, and 13.54 respectively. Therefore, we use Morfessor for segmentation in our proposed models.

Our models with additional decoders outperform baselines with only forward decoder. To assess the impact of different decoders and knowledge distillation approaches, we compare our model to various variations. The KD/RKD models with a backward decoder (BD) surpass the baseline in all three translation tasks. For English-to-Manipuri with Morfessor, the KD model achieves a BLEU score of 15.13, while the RKD model achieves a higher score of 15.76. Similarly, for English-to-Tamil with Morfessor, the KD model attains a score of 10.18, whereas the RKD model outperforms it with a score of 10.65. Similarly, for English-to-Marathi with Morfessor, the KD model attains a score of 14.02, whereas the RKD model outperforms it with a score of 14.46. Additionally, incorporating a morphological decoder (MD) enhances the performance of the baseline. The KD model with MD achieves BLEU scores of 14.79, 9.61 and 13.92 respectively, while the RKD model with MD achieves scores of 15.21, 9.86 and 14.12 respectively. Combining both backward decoder (BD) and morphological decoder (MD) further improves the score.

Our proposed models are competitive with related works. Comparing our proposed KD/RKD models with related works, we observe competitive performance. The Twin Networks approach, incorporating future information, achieves BLEU scores of 15.02 for English-to-Manipuri, 10.01 for English-to-Tamil and 13.85 for English-to-Marathi. ABDNMT, another model incorporating future information, achieves scores of 14.82, 9.75, and 14.01 for the respective tasks. SEER Forcing, employing the Seer forcing technique, achieves scores of 15.20, 10.22 and 14.05. Factored Translation, which augments class information, attains scores of 14.61, 9.43, and 13.52 for English-to-Manipuri and English-to-Tamil and English-to-Marathi, respectively.

As the morphological decoder has knowledge of morphological class information of the next word and the preceding words, we hope that the forward decoder can better predict the next word by distilling knowledge from the morphological decoder. In order to gain insights on whether the empirical usefulness comes from using morphological decoder, we perform two ablation tests. For “Gaussian Noise,” the backward probability distribution are randomly sampled from a Gaussian distribution, therefore the forward states are trained to predict white noise. For “AR,” the backward probability distribution are set to zero inspired by Merity et al. [23]. Results in Table 4 shows that the information included in the morphological states is indeed useful for obtaining a significant improvement.

In our approach, we have integrated the classes of the previous tokens \(class(y_{\lt t})\) and the upcoming token \(class(y_{t})\) into the morphological decoder. After conducting experiments, we have found that incorporating the class information for the next token \(class(y_{t})\) yields a modest improvement in performance compared to solely including the class information for the preceding tokens \(class(y_{\le t})\), as demonstrated in Table 5. This implies that accounting for the class of the upcoming token can provide more valuable contextual information for the model to generate the next token.

One of the key contributions of our approach is the use of a backward decoder, which possesses knowledge of future target information. We expect this knowledge to be transferred to the forward decoder through knowledge distillation.

Our experimental results, presented in both Table 3 and Table 5, clearly demonstrate the superiority of our approach over the baseline. In particular, we observe remarkable enhancements in performance when employing knowledge distillation with the backward decoder.

Interestingly, our results also show that the backward decoder outperforms the morphological decoder, indicating that our approach is able to leverage the additional information provided by the backward decoder to improve overall performance. Taken together, our findings suggest that the use of a backward decoder and knowledge distillation can significantly enhance the performance of neural language models.

In the context of knowledge distillation with the backward decoder, we conducted an investigation into various approaches for estimating reliability, and the results are presented in Table 6. Initially, we attempted to assign a binary reliability weight, represented by \(\beta _t\), with values of either 0 or 1, to indicate reliability or unreliability discretely. However, we found that this approach did not yield satisfactory results. As a result, we explored alternative approaches, including predicting the reliability weight using the hidden state \(\overleftarrow{h_t}\), by passing it through a linear layer and a sigmoid activation function. However, this approach also did not perform well.

Finally, we attempted to compute a continuous weight as the inverse of the cross-entropy between the ground truth and the predicted probability. This approach yielded the best results, outperforming the previous two approaches. The continuous weight allowed for more nuanced estimation of reliability, as opposed to a binary or discrete classification. In conclusion, we found that computing a continuous weight as the inverse of the cross-entropy between the ground truth and the predicted probability was the most effective approach for estimating reliability in knowledge distillation with the backward decoder.

As part of our evaluation process, we conducted a thorough analysis of the impact of our proposed method on sentence length in terms of the number of words with En\(\rightarrow\)Mni dataset. We achieved this by dividing the test set into four distinct groups based on the length of the target sentences. The results of our analysis, which are presented in Figure 5, demonstrated that the performance of our proposed method was dependent on the length of the target sentences.

In particular, we observed that our method was less effective when the length of the target sentence was between 0–10 words, with only minimal differences in BLEU scores compared to the baseline model. However, for sentences longer than 10 words, our proposed method showed a significant improvement in BLEU scores compared to the baseline transformer Trans.base model. This improvement in BLEU scores was even more pronounced as the length of the target translations increased, indicating that our method is particularly effective for longer sentences.

Overall, this analysis shows that our proposed method is a promising solution for improving the quality of translations, especially for longer sentences.

In this section, we will provide a comprehensive analysis of the training and decoding times, as well as the parameter sizes in the En\(\rightarrow\)Mni translation models. We present the results in Table 7.

The table compares three En\(\rightarrow\)Mni translation models, namely Baseline, RKD with BD, and RKD with MD. The models are evaluated based on their training and decoding times, as well as the size of their model parameters.

In terms of training and decoding times, the Baseline model has the lowest values, with a ratio of 1.0 for both training and decoding times. The RKD models, on the other hand, have a training time ratio of 1.7 and a decoding time ratio of 1.0.

When it comes to the size of the model parameters, baseline Transformer model have a smaller number of parameters with 13.6 million, and RKD models with 16.8 million.

Based on these findings, it is observed that the RKD models necessitate more time for training than the Baseline model, and they also involve extra parameters. However, it is noteworthy the backward decoder and morphological decoder are only used in training time and discarded during inference time. It makes the proposed RKD models match the inference time of the baseline Transformer model.

While this paper adopts a theoretical focus by using smaller, domain-specific datasets to train neural machine translation (NMT) models, it is also essential to consider the practical implications of scaling the approach to high-resource settings. To this end, we extend the English-Tamil dataset by incorporating additional publicly available corpora, including JW300, NLP Contributions (nlpc), OpenSubtitles, TED2020, and WikiMatrix, as provided in Nakazawa et al. [24]. The aggregated dataset contains a total of 1.35 million parallel sentences, providing a more larger resource for robust model training.

In this expanded setting, the model architecture consists of six input and output layers with an embedding dimension of 1024. The inner feedforward layers have a dimensionality of 4096, and the multi-head attention modules utilize 16 heads in both the encoder and decoder layers. Training is performed using mini-batches of 8,000 source and target tokens, which helps maintain stable gradient updates and efficient use of computational resources. Since the WAT 2021 test set is in the domain of PMI, after training on expanded dataset, we fine-tune on the PMI dataset.

The results in Table 8 highlight the impact of different segmentation strategies on translation quality. While the baseline model with Byte Pair Encoding (BPE) achieves a BLEU score of 10.21, replacing BPE with linguistically motivated Morfessor segmentation improves performance, yielding a BLEU score of 11.54. This improvement can be attributed to Morfessor’s ability to capture meaningful morphological patterns, especially useful for agglutinative languages like Tamil. Furthermore, our proposed Reliable Knowledge Distillation (RKD) approach, which integrates a backward decoder and a morphological decoder, achieves a BLEU score of 13.04. This demonstrates the effectiveness of our method for morphologically rich agglutinative languages, outperforming both BPE and Morfessor-based baselines.