skip to main content
research-article

Self-weighted Robust LDA for Multiclass Classification with Edge Classes

Published: 22 December 2020 Publication History

Abstract

Linear discriminant analysis (LDA) is a popular technique to learn the most discriminative features for multi-class classification. A vast majority of existing LDA algorithms are prone to be dominated by the class with very large deviation from the others, i.e., edge class, which occurs frequently in multi-class classification. First, the existence of edge classes often makes the total mean biased in the calculation of between-class scatter matrix. Second, the exploitation of ℓ2-norm based between-class distance criterion magnifies the extremely large distance corresponding to edge class. In this regard, a novel self-weighted robust LDA with ℓ2,1-norm based pairwise between-class distance criterion, called SWRLDA, is proposed for multi-class classification especially with edge classes. SWRLDA can automatically avoid the optimal mean calculation and simultaneously learn adaptive weights for each class pair without setting any additional parameter. An efficient re-weighted algorithm is exploited to derive the global optimum of the challenging ℓ2,1-norm maximization problem. The proposed SWRLDA is easy to implement and converges fast in practice. Extensive experiments demonstrate that SWRLDA performs favorably against other compared methods on both synthetic and real-world datasets while presenting superior computational efficiency in comparison with other techniques.

References

[1]
Karim T. Abou-Moustafa, Fernando De La Torre, and Frank P. Ferrie. 2010. Pareto discriminant analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[2]
Wei Bian and Dacheng Tao. 2008. Harmonic mean for subspace selection. In Proceedings of the International Conference on Pattern Recognition.
[3]
Wei Bian and Dacheng Tao. 2011. Max-min distance analysis by using sequential SDP relaxation for dimension reduction. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5 (2011), 1037--1050.
[4]
C. L. Blake and C. J. Merz. 1998. UCI Repository of Machine Learning Databases. Department of Information and Computer Science, University of California, Irvine, CA.
[5]
Deng Cai, Xiaofei He, and Jiawei Han. 2008. SRDA: An efficient algorithm for large-scale discriminant analysis. IEEE Trans. Knowl. Data Eng. 20, 1 (2008), 1--12.
[6]
Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 3 (2011), 27.
[7]
Xiaojun Chang, Po-Yao Huang, Yi-Dong Shen, Xiaodan Liang, Yi Yang, and Alexander G. Hauptmann. 2018. RCAA: Relational context-aware agents for person search. In Proceedings of the European Conference on Computer Vision, Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.).
[8]
Xiaojun Chang, Feiping Nie, Zhigang Ma, Yi Yang, and Xiaofang Zhou. 2015. A convex formulation for spectral shrunk clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, Blai Bonet and Sven Koenig (Eds.).
[9]
Xiaojun Chang, Haoquan Shen, Sen Wang, Jiajun Liu, and Xue Li. 2014. Semi-supervised feature analysis for multimedia annotation by mining label correlation. In Proceedings of the 18th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining (Lecture Notes in Computer Science), Vincent S. Tseng, Tu Bao Ho, Zhi-Hua Zhou, Arbee L. P. Chen, and Hung-Yu Kao (Eds.), Vol. 8444. Springer, 74--85.
[10]
Xiaojun Chang and Yi Yang. 2017. Semisupervised feature analysis by mining correlations among multiple tasks. IEEE Trans. Neural Netw. Learn. Syst. 28, 10 (2017), 2294--2305.
[11]
Xiaojun Chang, Yaoliang Yu, Yi Yang, and Eric P. Xing. 2016. They are not equally reliable: Semantic event search using differentiated concept classifiers. In Proceedings of the Conference on Computer Vision and Pattern Recognition.
[12]
De Cheng, Xiaojun Chang, Li Liu, Alexander G. Hauptmann, Yihong Gong, and Nanning Zheng. 2017. Discriminative dictionary learning with ranking metric embedded for person re-identification. In Proceedings of the International Joint Conference on Artificial Intelligence, Carles Sierra (Ed.).
[13]
De Cheng, Yihong Gong, Xiaojun Chang, Weiwei Shi, Alexander G. Hauptmann, and Nanning Zheng. 2018. Deep feature learning via structured graph Laplacian embedding for person re-identification. Pattern Recog. 82 (2018), 94--104.
[14]
C. Y. Chork and Peter J. Rousseeuw. 1992. Integrating a high-breakdown option into discriminant analysis in exploration geochemistry. J. Geochem. Explor. 43, 3 (1992), 191--203.
[15]
Franca Debole and Fabrizio Sebastiani. 2005. An analysis of the relative hardness of Reuters-21578 subsets. J. Amer. Soc. Inf. Sci. Technol. 56, 6 (2005), 584--596.
[16]
Mark Fanty and Ronald Cole. 1991. Spoken letter recognition. In Proceedings of the Conference on Neural Information Processing Systems.
[17]
Dewan Md Farid, Li Zhang, Chowdhury Mofizur Rahman, M. Alamgir Hossain, and Rebecca Strachan. 2014. Hybrid decision tree and Naïve Bayes classifiers for multi-class classification tasks. Exp. Syst. Applic. 41, 4 (2014), 1937--1946.
[18]
Keinosuke Fukunaga. 2013. Introduction to Statistical Pattern Recognition. Academic Press.
[19]
Athinodoros S. Georghiades, Peter N. Belhumeur, and David J. Kriegman. 2001. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intell. 23, 6 (2001), 643--660.
[20]
Chen Gong, Dacheng Tao, Xiaojun Chang, and Jian Yang. 2019. Ensemble teaching for hybrid label propagation. IEEE Trans. Cybern. 49, 2 (2019), 388--402.
[21]
Junwei Han, Le Yang, Dingwen Zhang, Xiaojun Chang, and Xiaodan Liang. 2018. Reinforcement cutting-agent learning for video object segmentation. In Proceedings of the Conference on Computer Vision and Pattern Recognition.
[22]
Xuming He and Wing K. Fung. 2000. High breakdown estimation for multiple populations with applications to discriminant analysis. J. Multivar. Anal. 72, 2 (2000), 151--162.
[23]
Geoffrey Holmes, Bernhard Pfahringer, Richard Kirkby, Eibe Frank, and Mark Hall. 2002. Multiclass alternating decision trees. In Proceedings of the European Conference on Machine Learning.
[24]
Mia Hubert and Katrien Van Driessen. 2004. Fast and robust discriminant analysis. Computat. Stat. Data Anal. 45, 2 (2004), 301--320.
[25]
Alexandros Iosifidis, Anastasios Tefas, and Ioannis Pitas. 2013. On the optimal class representation in linear discriminant analysis. IEEE Trans. Neural Netw. Learn. Syst. 24, 9 (2013), 1491--1497.
[26]
Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995. Elsevier, 331--339.
[27]
Zhihui Li, Feiping Nie, Xiaojun Chang, Liqiang Nie, Huaxiang Zhang, and Yi Yang. 2018. Rank-constrained spectral clustering with flexible embedding. IEEE Trans. Neural Netw. Learn. Syst. 29, 12 (2018), 6073--6082.
[28]
Zhihui Li, Feiping Nie, Xiaojun Chang, and Yi Yang. 2017. Beyond trace ratio: Weighted harmonic mean of trace ratios for multiclass discriminant analysis. IEEE Trans. Knowl. Data Eng. 29, 10 (2017), 2100--2110.
[29]
Yang Liu, Quanxue Gao, Xinbo Gao, and Ling Shao. 2018. L2,1-norm discriminant manifold learning. IEEE Access 6 (2018), 40723--40734.
[30]
Marco Loog, R. P. W. Duin, and Reinhold Haeb-Umbach. 2001. Multiclass linear dimension reduction by weighted pairwise Fisher criteria. IEEE Trans. Pattern Anal. Mach. Intell.7 (2001), 762--766.
[31]
Minnan Luo, Xiaojun Chang, Liqiang Nie, Yi Yang, Alexander G. Hauptmann, and Qinghua Zheng. 2018. An adaptive semisupervised feature analysis for video semantic recognition. IEEE Trans. Cybern. 48, 2 (2018), 648--660.
[32]
Minnan Luo, Feiping Nie, Xiaojun Chang, Yi Yang, Alexander G. Hauptmann, and Qinghua Zheng. 2016. Avoiding optimal mean robust PCA/2DPCA with non-greedy l1-norm maximization. In Proceedings of the International Joint Conference on Artificial Intelligence, Subbarao Kambhampati (Ed.).
[33]
Minnan Luo, Feiping Nie, Xiaojun Chang, Yi Yang, Alexander G. Hauptmann, and Qinghua Zheng. 2018. Adaptive unsupervised feature selection with structure regularization. IEEE Trans. Neural Netw. Learn. Syst. 29, 4 (2018), 944--956.
[34]
R. Douglas Martin and Víctor J. Yohai. 2006. Robust Statistics: Theory and Methods. Wiley.
[35]
S. Nayar, Sammeer A. Nene, and Hiroshi Murase. 1996. Columbia Object Image Library (Coil 100). Technical Report CUCS-006-96, Department of Computer Science, Columbia University.
[36]
Sameer A. Nene, Shree K. Nayar, Hiroshi Murase, et al. 1996. Columbia Object Image Library (Coil 20). Technical Report CUCS-006-96, Department of Computer Science, Columbia University.
[37]
Liqiang Nie, Luming Zhang, Yan Yan, Xiaojun Chang, Maofu Liu, and Ling Shaoling. 2017. Multiview physician-specific attributes fusion for health seeking. IEEE Trans. Cybern. 47, 11 (2017), 3680--3691.
[38]
Chong Peng, Jie Cheng, and Qiang Cheng. 2016. A supervised learning model for high-dimensional and large-scale data. ACM Trans. Intell. Syst. Technol. 8, 2 (2016), 1--23.
[39]
Chong Peng and Qiang Cheng. 2019. Discriminative ridge machine: A classifier for high-dimensional data or imbalanced data. arXiv preprint arXiv:1904.07496 (2019).
[40]
C. Radhakrishna Rao. 1948. The utilization of multiple measurements in problems of biological classification. J. Royal Stat. Soc. 10, 2 (1948), 159--203.
[41]
Terence Sim, Simon Baker, and Maan Bsat. 2002. The CMU pose, illumination, and expression (PIE) database. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition.
[42]
Gilbert W. Stewart. 2001. Eigensystem. Society for Industrial 8 Applied Mathematics.
[43]
Dacheng Tao, Xuelong Li, Xindong Wu, and Stephen J. Maybank. 2009. Geometric mean for subspace selection. IEEE Trans. Pattern Anal. Mach. Intell. 31, 2 (2009), 260--274.
[44]
Hao Wang, Yuanyuan Fan, Baofu Fang, and Shuanglu Dai. 2018. Generalized linear discriminant analysis based on Euclidean norm for gait recognition. J. Mach. Learn. Cybern. 9, 4 (2018), 569--576.
[45]
Sen Wang, Xiaojun Chang, Xue Li, Quan Z. Sheng, and Weitong Chen. 2016. Multi-task support vector machines for feature selection with shared knowledge discovery. Sig. Proc. 120 (2016), 746--753.
[46]
Jie Wen, Xiaozhao Fang, Jinrong Cui, Lunke Fei, Ke Yan, Yan Chen, and Yong Xu. 2019. Robust sparse linear discriminant analysis. IEEE Trans. Circ. Syst. Vid. Technol. 29, 2 (2019), 390--403.
[47]
Liping Xie, Dacheng Tao, and Haikun Wei. 2017. Joint structured sparsity regularized multiview dimension reduction for video-based facial expression recognition. ACM Trans. Intell. Syst. Technol. 8, 2 (2017), 28.
[48]
Haoyi Xiong, Jinghe Zhang, Yu Huang, Kevin Leach, and Laura E. Barnes. 2017. Daehr: A discriminant analysis framework for electronic health record data and an application to early detection of mental health disorders. ACM Trans. Intell. Syst. Technol. 8, 3 (2017), 47.
[49]
Bo Xu, Kaizhu Huang, and Cheng-Lin Liu. 2010. Dimensionality reduction by minimal distance maximization. In Proceedings of the International Conference on Pattern Recognition.
[50]
Xiaowei Xue, Feiping Nie, Sen Wang, Xiaojun Chang, Bela Stantic, and Min Yao. 2017. In Proceedings of the AAAI Conference on Artificial Intelligence, Satinder P. Singh and Shaul Markovitch (Eds.).
[51]
Sharipah Soaad Syed Yahaya, Yai-Fung Lim, Hazlina Ali, and Zurni Omar. 2016. Robust linear discriminant analysis with automatic trimmed mean. J. Telecommun. Electron. Comput. Eng. 8, 10 (2016), 1--3.
[52]
Caixia Yan, Minnan Luo, Huan Liu, Zhihui Li, and Qinghua Zheng. 2018. Top-k multi-class SVM using multiple features. Inf. Sci. 432 (2018), 479--494.
[53]
Jieping Ye. 2007. Least squares linear discriminant analysis. In Proceedings of the International Conference on Machine Learning.
[54]
Jieping Ye, Ravi Janardan, Qi Li, and Haesun Park. 2006. Feature reduction via generalized uncorrelated linear discriminant analysis. IEEE Trans. Knowl. Data Eng. 18, 10 (2006), 1312--1322.
[55]
Jieping Ye and Tao Xiong. 2006. Null space versus orthogonal linear discriminant analysis. In Proceedings of the 23rd International Conference on Machine Learning. 1073--1080.
[56]
Yaoliang Yu, Jiayan Jiang, and Liming Zhang. 2011. Distance metric learning by minimal distance maximization. Pattern Recog. 44, 3 (2011), 639--649.
[57]
Sen Yuan, Xia Mao, and Lijiang Chen. 2017. Multilinear spatial discriminant analysis for dimensionality reduction. IEEE Trans. Image Proc. 26, 6 (2017), 2669--2681.
[58]
Liang Zhang, Lindsay B. Jack, and Asoke K. Nandi. 2005. Extending genetic programming for multi-class classification by combining k-nearest neighbor. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing.
[59]
Tianhao Zhang, Dacheng Tao, and Jie Yang. 2008. Discriminative locality alignment. In Proceedings of the European Conference on Computer Vision. Springer, 725--738.
[60]
Xiaoqin Zhang, Mingyu Fan, Di Wang, Peng Zhou, and Dacheng Tao. 2020. Top-k feature selection framework using robust 0-1 integer programming. IEEE Trans. Neural Netw. Learn. Syst. (2020).
[61]
Xu-Yao Zhang and Cheng-Lin Liu. 2012. Confused distance maximization for large category dimensionality reduction. In Proceedings of the International Conference on Frontiers in Handwriting Recognition.
[62]
Yu Zhang and Dit-Yan Yeung. 2010. Worst-case linear discriminant analysis. In Proceedings of the Conference on Neural Information Processing Systems.
[63]
Haifeng Zhao, Siqi Wang, and Zheng Wang. 2018. Multiclass classification and feature selection based on least squares regression with large margin. Neural Computat. 30, 10 (2018), 2781--2804.
[64]
Haifeng Zhao, Zheng Wang, and Feiping Nie. 2019. A new formulation of linear discriminant analysis for robust dimensionality reduction. IEEE Trans. Knowl. Data Eng. 31, 4 (2019), 629--640.
[65]
Xiaowei Zhao, Feiping Nie, Sen Wang, Jun Guo, Pengfei Xu, and Xiaojiang Chen. 2017. Unsupervised 2D dimensionality reduction with adaptive structure learning. Neural Computat. 29, 5 (2017), 1352--1374.
[66]
Yang Zhou and Shiliang Sun. 2016. Manifold partition discriminant analysis. IEEE Trans. Cybern. 47, 4 (2016), 830--840.
[67]
Lei Zhu, Zi Huang, Xiaojun Chang, Jingkuan Song, and Heng Tao Shen. 2017. Exploring consistent preferences: Discrete hashing with pair-exemplar for scalable landmark search. In Proceedings of the ACM International Conference on Multimedia, Qiong Liu, Rainer Lienhart, Haohong Wang, Sheng-Wei “Kuan-Ta” Chen, Susanne Boll, Yi-Ping Phoebe Chen, Gerald Friedland, Jia Li, and Shuicheng Yan (Eds.).

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology
ACM Transactions on Intelligent Systems and Technology  Volume 12, Issue 1
Regular Papers
February 2021
280 pages
ISSN:2157-6904
EISSN:2157-6912
DOI:10.1145/3436534
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 December 2020
Accepted: 01 August 2020
Revised: 01 August 2020
Received: 01 December 2019
Published in TIST Volume 12, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Robust linear discriminant analysis
  2. dimension reduction
  3. edge class
  4. multi-class classification

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

  • Innovation Research Team of Ministry of Education (IRT 17R86)
  • National Key Research and Development Program of China
  • National Nature Science Foundation of China
  • National Natural Science Foundation of China
  • China Scholarship Council
  • Innovative Research Group of the National Natural Science Foundation of China
  • Zhejiang Provincial Natural Science Foundation

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)105
  • Downloads (Last 6 weeks)13
Reflects downloads up to 15 Sep 2024

Other Metrics

Citations

Cited By

View all

View Options

Get Access

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media