Group Lasso Estimation of High-dimensional Covariance Matrices
Authors: 
Jérémie Bigot, Rolando J. Biscay, Jean-Michel Loubes, Lillian Muñiz-AlvarezAuthors Info & Claims
Jérémie Bigot, Rolando J. Biscay, Jean-Michel Loubes, Lillian Muñiz-AlvarezAuthors Info & ClaimsPages 3187 - 3225
Abstract
In this paper, we consider the Group Lasso estimator of the covariance matrix of a stochastic process corrupted by an additive noise. We propose to estimate the covariance matrix in a high-dimensional setting under the assumption that the process has a sparse representation in a large dictionary of basis functions. Using a matrix regression model, we propose a new methodology for high-dimensional covariance matrix estimation based on empirical contrast regularization by a group Lasso penalty. Using such a penalty, the method selects a sparse set of basis functions in the dictionary used to approximate the process, leading to an approximation of the covariance matrix into a low dimensional space. Consistency of the estimator is studied in Frobenius and operator norms and an application to sparse PCA is proposed.
References
[1]
Anestis Antoniadis, Jeremie Bigot, and Theofanis Sapatinas. Wavelet estimators in nonparametric regression: A comparative simulation study. Journal of Statistical Software, 6(6):1-83, 6 2001.
[2]
Francis R. Bach. Consistency of the group lasso and multiple kernel learning. J. Mach. Learn. Res., 9:1179-1225, 2008.
[3]
Peter J. Bickel and Elizaveta Levina. Regularized estimation of large covariance matrices. Ann. Statist., 36(1):199-227, 2008a.
[4]
Peter J. Bickel and Elizaveta Levina. Covariance regularization by thresholding. Ann. Statist., 36 (6):2577-2604, 2008b.
[5]
Peter J. Bickel, Ya'acov Ritov, and Alexandre B. Tsybakov. Simultaneous analysis of lasso and Dantzig selector. Ann. Statist., 37(4):1705-1732, 2009.
[6]
Jérémie Bigot, Rolando J. Biscay, Jean-Michel Loubes, and Lilian Muñiz Alvarez. Nonparametric estimation of covariance functions by model selection. Electronic Journal of Statistics, 4:822-855, 2010.
[7]
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, Cambridge, 2004. ISBN 0-521-83378-7.
[8]
Noel A. C. Cressie. Statistics for Spatial Data. Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics. John Wiley & Sons Inc., New York, 1993.
[9]
Alexandre d'Aspremont, Francis Bach, and Laurent El Ghaoui. Optimal solutions for sparse principal component analysis. J. Mach. Learn. Res., 9:1269-1294, 2008.
[10]
Kenneth R. Davidson and Stanislaw J. Szarek. Local operator theory, random matrices and Banach spaces. In Handbook of the Geometry of Banach Spaces, Vol. I, pages 317-366. North-Holland, Amsterdam, 2001.
[11]
Chandler Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. III. SIAM J. Numer. Anal., 7:1-46, 1970.
[12]
Noureddine El Karoui. Operator norm consistent estimation of large-dimensional sparse covariance matrices. Ann. Statist., 36(6):2717-2756, 2008.
[13]
Jianquin Fan, Yingying Fan, and Jinchi Lv. High dimensional covariance matrix estimation using a factor model. Journal of Econometrics, 147:186-197, 2008.
[14]
Junzhou Huang and Tong Zhang. The benefit of group sparsity. Ann. Statist., 38(4):1978-2004, 2010. ISSN 0090-5364.
[15]
Iain M. Johnstone. On the distribution of the largest eigenvalue in principal components analysis. Ann. Statist., 29(2):295-327, 2001.
[16]
Iain M. Johnstone and Arthur Y. Lu. On consistency and sparsity for principal components analysis in high dimensions. Journal of the American Statistical Association, 104(486):682-693, June 2009.
[17]
Andre G. Journel. Kriging in terms of projections. J. Internat. Assoc. Mathematical Geol., 9(6): 563-586, 1977.
[18]
Clifford Lam and Jianqing Fan. Sparsistency and rates of convergence in large covariance matrix estimation. Ann. Statist., 37(6B):4254-4278, 2009.
[19]
Elizaveta Levina, Adam Rothman, and Ji Zhu. Sparse estimation of large covariance matrices via a nested Lasso penalty. Ann. Appl. Stat., 2(1):245-263, 2008.
[20]
Karim Lounici. Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators. Electron. J. Stat., 2:90-102, 2008.
[21]
Karim Lounici, Massimiliano Pontil, Alexandre B. Tsybakov, and Sara van de Geer. Taking advantage of sparsity in multi-task learning. COLT, 2009.
[22]
Karim Lounici, Massimiliano Pontil, Alexandre B. Tsybakov, and Sara van de Geer. Oracle Inequalities and Optimal Inference under Group Sparsity. Ann. Statist., to be published., 2011.
[23]
Shahar Mendelson and Alain Pajor. On singular values of matrices with independent rows. Bernoulli, 12(5):761-773, 2006.
[24]
Yuval Nardi and Alessandro Rinaldo. On the asymptotic properties of the group lasso estimator for linear models. Electron. J. Stat., 2:605-633, 2008.
[25]
Adam J. Rothman, Peter J. Bickel, Elizaveta Levina, and Ji Zhu. Sparse permutation invariant covariance estimation. Electron. J. Stat., 2:494-515, 2008.
[26]
Mark Schmidt, KevinMurphy, Glenn Fung, and Rómer Rosales. Structure learning in random fields for heart motion abnormality detection (addendum). CVPR08, 2008.
[27]
Michael L. Stein. Interpolation of Spatial Data. Some Theory for Kriging. Springer Series in Statistics. New York, NY: Springer. xvii, 247 p., 1999.
[28]
Christopher K. Wikle and Noel Cressie. A dimension-reduced approach to space-time Kalman filtering. Biometrika, 86(4):815-829, 1999. ISSN 0006-3444.
[29]
Hui Zou, Trevor Hastie, and Robert Tibshirani. Sparse principal component analysis. J. Comput. Graph. Statist., 15(2):265-286, 2006.
- Group Lasso Estimation of High-dimensional Covariance Matrices
Recommendations
Sparse group lasso and high dimensional multinomial classification
The sparse group lasso optimization problem is solved using a coordinate gradient descent algorithm. The algorithm is applicable to a broad class of convex loss functions. Convergence of the algorithm is established, and the algorithm is used to ...
Group Fused Lasso
Proceedings of the 23rd International Conference on Artificial Neural Networks and Machine Learning ICANN 2013 - Volume 8131We introduce the Group Total Variation (GTV) regularizer, a modification of Total Variation that uses the ℓ2,1 norm instead of the ℓ1 one to deal with multidimensional features. When used as the only regularizer, GTV can be applied jointly with ...
A well-conditioned and sparse estimation of covariance and inverse covariance matrices using a joint penalty
We develop a method for estimating well-conditioned and sparse covariance and inverse covariance matrices from a sample of vectors drawn from a sub-Gaussian distribution in high dimensional setting. The proposed estimators are obtained by minimizing the ...
Comments
Information & Contributors
Information
Published In
Publisher
JMLR.org
Publication History
Published: 01 November 2011
Published in JMLR Volume 12
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 207Total Downloads
- Downloads (Last 12 months)46
- Downloads (Last 6 weeks)4
Reflects downloads up to 03 Jan 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in