Defects of convolutional decoder networks in frequency representation
Article No.: 1406, Pages 33758 - 33791
Abstract
In this paper, we prove the representation defects of a cascaded convolutional decoder1 network, considering the capacity of representing different frequency components of an input sample. We conduct the discrete Fourier transform on each channel of the feature map in an intermediate layer of the decoder network. Then, we extend the 2D circular convolution theorem to represent the forward and backward propagations through convolutional layers in the frequency domain. Based on this, we prove three defects in representing feature spectrums. First, we prove that the convolution operation, the zero-padding operation, and a set of other settings all make a convolutional decoder network more likely to weaken high-frequency components. Second, we prove that the upsampling operation generates a feature spectrum, in which strong signals repetitively appear at certain frequencies. Third, we prove that if the frequency components in the input sample and frequency components in the target output for regression have a small shift, then the decoder usually cannot be effectively learned.
References
[1]
Amjad, R. A. and Geiger, B. C. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence, 42(9):2225-2239, 2019.
[2]
Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3319-3327, 2017.
[3]
Deng, H., Ren, Q., Chen, X., Zhang, H., Ren, J., and Zhang, Q. Discovering and explaining the representation bottleneck of dnns. In International Conference on Learning Representations, 2022.
[4]
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018.
[5]
Jain, A. K. Fundamentals of digital image processing. Prentice-Hall, Inc., 1989.
[6]
Kreutz-Delgado, K. The complex gradient operator and the cr-calculus. arXiv preprint arXiv:0906.4835, 2009.
[7]
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
[8]
Le, Y. and Yang, X. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[9]
Li, X., Chen, S., and Yang, J. Understanding the disharmony between weight normalization family and weight decay. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[10]
Lin, Y., Jiang, H., et al. Bandlimiting neural networks against adversarial attacks. arXiv preprint arXiv:1905.12797, 2019.
[11]
Luo, T., Ma, Z., Xu, Z.-Q. J., and Zhang, Y. Theory of the frequency principle for general deep neural networks. arXiv preprint arXiv:1906.09235, 2019.
[12]
Ma, C., Wu, L., et al. Machine learning from a continuous viewpoint, i. Science China Mathematics, 63(11):2233-2266, 2020.
[13]
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2019.
[14]
Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y., and Courville, A. On the spectral bias of neural networks. In International Conference on Machine Learning, pp. 5301-5310. PMLR, 2019.
[15]
Reinhard, H. and Fatih, F. Y. Early stopping in deep networks: Double descent and how to eliminate it. In International Conference on Learning Representations, 2020.
[16]
Ruderman, D. L. The statistics of natural images. Network: computation in neural systems, 5(4):517, 1994.
[17]
Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017.
[18]
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[19]
Sundararajan, D. The discrete Fourier transform: theory, algorithms and applications. World Scientific, 2001.
[20]
Tishby, N. and Zaslavsky, N. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
[21]
Tse, D. and Viswanath, P. Fundamentals of wireless communication. Cambridge university press, 2005.
[22]
Van Laarhoven, T. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350, 2017.
[23]
Wang, H., Wu, X., Huang, Z., and Xing, E. P. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8684-8694, 2020.
[24]
Wolchover, N. and Reading, L. New theory cracks open the black box of deep learning. Quanta Magazine, 3, 2017.
[25]
Xu, Z.-Q. J., Zhang, Y., Luo, T., Xiao, Y., and Ma, Z. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523, 2019a.
[26]
Xu, Z.-Q. J., Zhang, Y., and Xiao, Y. Training behavior of deep neural network in frequency domain. In International Conference on Neural Information Processing, pp. 264-274. Springer, 2019b.
[27]
Yin, D., Gontijo Lopes, R., Shlens, J., Cubuk, E. D., and Gilmer, J. A fourier perspective on model robustness in computer vision. Advances in Neural Information Processing Systems, 32, 2019.
Recommendations
Sphere decoder with box optimisation for faster‐than‐Nyquist non‐orthogonal frequency division multiplexing
In 1975, J. E. Mazo showed the potential faster‐than‐Nyquist (FTN) gain of the single‐carrier binary signal. If the inter‐symbol interference is eliminated by an optimal detector, FTN single‐carrier binary signal can transmit 24.7% more bits than the ...
Residual Frequency Offset Compensation-Embedded Turbo Decoder
Proceedings of the 2007 IEEE Wireless Communications and Networking ConferenceWe propose a modified iterative turbo decoder, which inherently compensates the residual frequency offset remained in the decoder input. Based on the recently proposed phase offset compensation scheme which embeds phase compensation into iterative turbo ...
Non‐orthogonal frequency division multiplexing based on sparse representation
This study proposes a sparse non‐orthogonal frequency division multiplexing (S‐NOFDM) based on sparse representation to improve the spectral efficiency of orthogonal frequency division multiplexing (OFDM). The subcarriers of S‐NOFDM are generated from a ...
Comments
Information & Contributors
Information
Published In
July 2023
43479 pages
Copyright © 2023.
Publisher
JMLR.org
Publication History
Published: 23 July 2023
Qualifiers
- Research-article
- Research
- Refereed limited
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 16 Oct 2024