Towards Robust Speech Super-resolution

A. Experimental Setup

We evaluate our model on two datasets, TIMIT [12] and VCTK [48]. TIMIT is a standard corpus containing speech recordings from 630 speakers with a 16 kHz sampling rate. From the training part of the corpus, we choose 4620 utterances as the training dataset, and 1153 utterances as the validation set. We select a subset of the TIMIT core test set for test purposes, which consists of 192 utterances from 24 speakers that are not included in the training and validation datasets, thus enabling us to assess the generalization ability to untrained speakers. The VCTK corpus contains 44 hours of speech recordings from 108 speakers with a 48 kHz sampling rate. For a fair comparison, we follow the task design of [22] and [25]. The first task uses speech data of one specific speaker (speaker p225). The other task is multi-speaker, for which we train and test using the whole corpus. Following the description in [25], we split the data to 88%, 6%, and 6% for training, validation and testing purposes. We also make sure that there is no speaker overlap between training, validation and testing for the multi-speaker task. LR signals are obtained by first applying a Chebynov type I low-pass filter and then subsampling. A silence filter that discards samples below an energy threshold of 0.05 is performed to stabilize training and ensure faster convergence.

For preprocessing, all the utterances are first resampled to a 16 kHz sampling rate if their original sampling rate is higher than 16 kHz. Then each utterance is normalized to zero mean and unit variance. Note this is different from our preliminary version [50], where we apply a uniform normalization (rescaling each utterance to the range −1.0 to 1.0). We observe that mean and variance normalization (MVN) improves cross-corpus generalization. We divide each utterance into frames of 2048 samples (128 ms), and with an overlap of 1024 samples between consecutive frames. LR inputs are upsampled using cubic spline interpolation so that input and desired output signals for our CNN have the same length. During the reconstruction stage, we combine consecutive frames using the OLA method.

We evaluate the SR performance with three objective metrics: SNR, LSD, and PESQ for wideband speech [5]. SNR is a time-domain metric, defined as,

LSD, a frequency-domain metric, measures the logarithmic distance between two magnitude spectra in dB.

When the two spectra are the same, LSD will be 0 dB, the smallest possible distance. Given a reference and a degraded audio signal, PESQ for wideband speech is a standard metric of perceptual speech quality with a value range from 1.04 to 4.64. Higher PESQ indicates better listening quality.

B. Comparison Models

We compare with four other deep SR models described in Section II. The first baseline is the spectral domain model by Li and Lee [23], referred to as DNN-BWE. This model takes 9 frames (4 preceding and 4 succeeding) as the input and predicts the current STFT frame. We follow the implementation details in the original work, which uses 256-sample frames with a frame shift of 128 samples, and a 4-layer DNN with 2048 hidden units for training. The second baseline is by Abel and Fingscheidt (denoted as DNN-Cepstral) [1], [2], and we follow the original description by using a 256-sample frame length and a 128-sample frame shift for narrowband speech. Their DNN has 3 hidden layers with 256 units in each. The third comparison model is the waveform-based model proposed by Kuleshov et al. [22], denoted as AudioUNet. Following their default setting in the publicly available code¹, we set the number of filters in the encoding layers to 128, 256, 512, and 512, and the filter size to 65, 33, 17, and 9. The setting for decoder layers is similar except for the reverse order and doubled filter size. AudioUNet operates on 2048-sample segments, with a 50% overlap between consecutive segments. The last baseline is TFNet by Lim et al. [25], which consists of two AudioUNets. But for both branches, the number of filters is halved to reduce parameters. To be consistent with the settings in their paper, a frame size of 8192 samples with a 75% overlap is employed for the network input.

For our proposed network, the kernel size is set to 11 for all convolutional layers. The number of channels for each encoding layer is set to 64, 64, 64, 128, 128, 128, 256, 256, and 256, and the decoding layers have the same numbers except in the reverse order. Our network is trained with a mini-batch size of 32 for 100 epochs, and optimized with the T-PCM loss. Dropout ratio is set to 0.2 for dropout layers. The Adam optimizer [20] with a learning rate of 0.0003 is used for stochastic gradient descent based optimization. The learning rate is halved if the loss has not improved for 3 consecutive epochs on the validation set. We add an early stopping criterion such that the training process stops if the validation loss has not improved for 6 successive epochs. For other deep SR baselines, the training setup follows their original descriptions.

C. Results and Comparisons

Table I presents the results of our proposed CNN, as well as the other baselines, on the TIMIT dataset which is downsampled to 8 kHz to create LR signals. The first row corresponds to the objective scores by applying cubic spline interpolation, which is a conventional signal processing baseline and outputs limited but stable improvement for SR. Our model improves over the cubic spline method by 4.7 dB in terms of SNR, and cuts LSD by 68.3%. PESQ is improved by nearly 1.1, which is a large improvement for speech quality. Compared with the other four deep learning baselines, we see consistent improvement over all three metrics. Specifically, compared with the best-performing baseline of TFNet, our model improves SNR by around 1.3 dB, LSD by 17.2%, and PESQ by 0.5. Also our model slightly improves over the AECNN baseline in terms of SNR and PESQ.

The VCTK results are reported in Table II. The ratio R is the downsampling factor, where R = 2 implies upsampling from 8 kHz to 16 kHz, and R = 4 represents upsampling from 4 kHz to 16 kHz. VCTK_S represents the single-speaker task, and VCTK_M represents the multi-speaker task. As shown in Table II, our model shows superior performance for both downsampling factors. For R = 2, compared with the spline baseline, our model improves SNR by over 3.0 dB, and cuts LSD to below 1.0 for both tasks. We also see a consistent improvement in PESQ. For R = 4, we observe similar improvements for all three metrics over the spline method. For instance, for the multi-speaker task, we see SNR boosted by around 4.7 dB, LSD cut to below 1.0, and PESQ increased by 0.5. Our model also consistently improves over other deep SR baselines. Compared with the strongest baseline of TFNet, our proposed model yields better SNR, LSD and PESQ scores for both tasks under the two downsampling factors. For the multispeaker task under R = 2, for example, SNR is improved by around 2.2 dB, LSD by 11.1%, and PESQ by around 0.2.

To examine the advantage of our proposed T-PCM loss function, we compare different loss functions on the TIMIT corpus. As shown in Table III, the time-domain losses L_MSE and L_MAE give good SNR values, but not LSD performance. By introducing spectral loss terms (L_F, L_RI, L_RI-MAG, L_TF, L_PCM and L_T-PCM), the PESQ values increase, indicating that spectral loss leads to better speech quality. Although L_F and L_RI have good LSD scores, very low SNR scores are observed. Although L_RI manifests both phase and magnitude, the real and imaginary parts are related as the cosine and sine, respectively, of the phase multiplied by the magnitude [52]. By including a magnitude term, L_RI-MAG is found to improve both phase and magnitude estimation over L_RI. Additionally, due to poor phase estimation, for L_F and L_RI we observe unwanted artifacts in reconstructed speech for certain signals obtained by decimating downsampling (see Section V-B). The L_TF and L_RI-MAG losses show balanced performance in terms of SNR, LSD, and PESQ, and artifacts are alleviated but still exist. For L_PCM and L_T-PCM, we do not hear the artifact in reconstructed speech.

Table IV compares the numbers of trainable parameters of our proposed model and the other baselines, which show that our model achieves strong performance with a relatively small number of parameters. Fig. 3 illustrates the output of our SR model on a sample TIMIT utterance ( “In wage negotiations, the industry bargains as a unit with a single union”). Comparing the spectrograms we can observe that missing high-frequency components in the LR spectrogram are recovered well by our model.

A. Corpus Channels

A speech corpus typically contains speech signals recorded in a fixed environment. Taking TIMIT for example, all recordings are collected in the same anechoic room with a single microphone. Although this setting guarantees a uniform quality, it likely introduces signal characteristics unique to the specific experimental setting, impeding the generalizability of the trained models on one corpus. The recording characteristic of a corpus is referred to as corpus channel [39]. To validate this analysis, we randomly choose two microphone impulse response (MIR) functions from Vintage Mics², and convolve them with TIMIT utterances. As shown in Fig. 4, the energy distributions over frequency are distinct for the same utterance when convolved with different MIRs. Table V further illustrates channel effects, where our model trained with original TIMIT utterances shows degraded performance when tested on utterances convolved with the two MIRs. The last row is the average result of testing separately on utterances convolved with 20 randomly picked MIRs.

To systematically investigate channel effects, we conduct cross-corpus experiments on four different databases using the deep SR models evaluated in the previous section. Experimental settings and DNN architectures are as described in Section IV-A and IV-B. The four datasets are TIMIT [12], Wall Street Journal (WSJ) [40], LIBRIspeech [36], and IEEE [19]. For the WSJ corpus, speakers read Wall Street Journal articles plus spontaneous dictations. Two sets of microphones are utilized for the recordings: a close-talking Sennheiser HMD414 and a secondary microphone which may vary. WSJ recordings are sampled at a 16 kHz sampling rate, and contain a small amount of background noise. For our experiments, we use 12736 utterances from 100 speakers to train, 1206 utterances from 10 speakers to validate, and 651 utterances from 8 speakers to test. LibriSpeech consists of 1000 hours of 16 kHz English speech recordings, which are derived from reading audiobooks in the LibriVox project. The recordings are collected from volunteers across the world, so LibriSpeech has various recording environments and thus contains diverse channels. In this paper, the LibriClean subset (denoted as LIBRI) of the LibriSpeech corpus is chosen for our experiments. From the LIBRI corpus, we select 28539 utterances for training, 2703 utterances for validation and 2620 utterances for testing, with no speaker overlap. IEEE contains 720 phonetically balanced English sentences uttered by a male speaker with a sampling frequency of 25 kHz. We randomly select 576 utterances for training, 72 utterances for validation and the remaining 72 utterances are reserved for testing purposes. For all datasets, utterances are first resampled to 16 kHz, and LR signals are generated at 8 kHz by applying the subsampling scheme.

Table VI summarizes the results of cross-corpus SR experiments. Each row represents one model trained on a particular dataset and tested on all four datasets. Each column shows the results on a specific dataset. As shown in the table, our model outperforms all the other baselines for all four datasets. As expected, the best performance is observed when training and testing are done on the same corpus. We observe that the generalization ability of each model differs with training dataset, and the models trained with datasets that contain diverse channels are more robust when testing on untrained corpora. Specifically, training with WSJ or LIBRI shows comparable objective scores when tested on untrained corpora. Training with IEEE or TIMIT, however, shows poor performance on untrained corpus channels. This observation is more obvious for AudiouNet, TFNet and the proposed network. For the spectral-domain models of DNN-BWE and DNN-Cepstral, the generalization advantage of WSJ and LIBRI mainly manifests in LsD and pEsQ. These models perform relatively poorly for the time-domain metrics of SNR, likely because they focus on magnitude optimization. In addition, we expect that training on multiple corpora should enhance robustness to the trained corpora. To verify this, we train our model by randomly selecting 10000 utterances from the training sets of the four datasets (TIMIT, WSJ, LIBRI and IEEE), and then test on their test sets. The results are given in the last row of Table VI, and show that this strategy yields good performance for all four datasets.

We remark that the size of the corpus does not seem to be a key factor for generalization. Although IEEE has only about a sixth of the utterances of the TIMIT dataset, the models trained on TIMIT do not display better generalization. Especially time-domain models (AudioUNet, TFNet, Proposed) trained with TIMIT perform even worse than trained with IEEE on untrained corpora. Another remark is that the models trained with WSJ and LIBRI have comparable robustness even though LIBRI contains more microphone channels. This may result from the fact that WsJ contains more background noise, which is another factor that affects cross-corpus generalization.

B. Downsampling Schemes

Most of the existing speech SR networks are trained with simulated datasets, where LR/HR pairs are generated by applying a specific downsampling scheme. In the real world, however, the pre-assumed downsampling scheme may not match the LR/HR relationship. Our experiments indicate DNN-based models are sensitive to different downsampling schemes, and this affects the generalization capability of supervised SR models.

We divide downsampling schemes into three categories. The first one is referred to as subsampling, which is the default setting of the MATLAB [30] downsample function. Subsampling decreases the sampling rate by discarding samples at fixed intervals. The second category is decimating, where one first applies a low-pass filter and then subsamples to acquire the desired LR signal. This is the default setting for the MATLAB decimate, resample functions and the SciPy [49] decimate function. The first two methods operate in the time domain, and the third category named FFT operates in the frequency domain. The FFT scheme transforms an HR signal to the Fourier domain, leaves out high-frequency parts above the cutoff frequency, and then transforms back to obtain the corresponding LR signal. This is the default setting of the resample function in SciPy.

Table VII provides a comparison of the three categories of downsampling schemes on TIMIT using the proposed model. In the first three rows, we use the model trained with one specific downsampling scheme to test on data obtained by all three downsampling schemes. we observe that the decimating scheme performs the worst when tested on untrained schemes, with a drastic drop in objective scores (even negative SNR values). Although we see a degradation for subsampling and FFT, the drop is not nearly as severe as decimating.

Subsampling is simple and efficient among the three schemes. However, according to the sampling theorem of Shannon [43], this method introduces an unwanted artifact (referred to as aliasing) during the downsampling process as there are components with frequencies higher than the Nyquist frequency. Decimating solves this problem by first applying a low-pass filter. By default decimating uses chebyshev Type I infinite impulse response filter of order 8 as the anti-aliasing filter in both MATLAB and SciPy package. We investigate two other low-pass filters (Butterworth and Bessel) for decimating. Experiments show that SR performance is highly affected by the type of low-pass filters. This indicates the models learned using decimating schemes carry unwanted characteristics of specific filters, which limit their application to realistic signals. The FFT method also avoids the aliasing phenomenon, and is a better choice for generalization purposes since it does not involve any filter. However, the computational expense is higher than the other two schemes.

It is important to develop a model that is robust against downsampling schemes. To achieve this, we introduce a random downsampling strategy: for each HR signal, the corresponding LR signal is generated by randomly picking one downsampling scheme from the three categories. By doing so, we make sure that models learn the essential features for SR, not the acoustic properties of specific downsampling techniques. The last row in Table VII provides the results when training with random downsampling. The results demonstrate that the model trained in this way is capable of producing satisfactory SR performance regardless of how LR signals are generated.