On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
Abstract
Many tasks in machine learning and signal processing can be solved by minimizing a convex function of a measure. This includes sparse spikes deconvolution or training a neural network with a single hidden layer. For these problems, we study a simple minimization method: the unknown measure is discretized into a mixture of particles and a continuous-time gradient descent is performed on their weights and positions. This is an idealization of the usual way to train neural networks with a large hidden layer. We show that, when initialized correctly and in the many-particle limit, this gradient flow, although non-convex, converges to global minimizers. The proof involves Wasserstein gradient flows, a by-product of optimal transport theory. Numerical experiments show that this asymptotic behavior is already at play for a reasonable number of particles, even in high dimension.
- Publication:
-
arXiv e-prints
- Pub Date:
- May 2018
- DOI:
- arXiv:
- arXiv:1805.09545
- Bibcode:
- 2018arXiv180509545C
- Keywords:
-
- Mathematics - Optimization and Control;
- Computer Science - Neural and Evolutionary Computing;
- Statistics - Machine Learning
- E-Print:
- Advances in Neural Information Processing Systems (NIPS), Dec 2018, Montr\'eal, Canada