research-article

A Constrained Competitive Swarm Optimizer With an SVM-Based Surrogate Model for Feature Selection

Authors:

Bach Hoai Nguyen,

Mengjie ZhangAuthors Info & Claims

IEEE Transactions on Evolutionary Computation, Volume 28, Issue 1

Pages 2 - 16

https://doi.org/10.1109/TEVC.2022.3197427

Published: 01 February 2024 Publication History

Abstract

Feature selection (FS) is an important data preprocessing technique that selects a small subset of relevant features to improve learning performance. However, it is also challenging due to its large search space. Recently, a competitive swarm optimizer (CSO) has shown promising results in FS because of its potential global search ability. The main idea of CSO is to select two solutions randomly and then let the loser (worse fitness) learn from the winner (better fitness). Although such a search mechanism provides a high population diversity, it is at risk of generating unqualified solutions since the winner’s quality is not guaranteed. In this work, we propose a constrained evolutionary mechanism for CSO, which verifies the quality of all the particles and lets the infeasible (unqualified) solutions learn from the feasible (qualified) ones. We also propose a novel local search and a size-change operator that guide the population to search for smaller feature subsets with similar or better classification performance. A surrogate model, based on support vector machines, is proposed to assist both local search and the size-change operator to explore a massive number of potential feature subsets without requiring excessive computational resource. Results on 24 real-world datasets show that the proposed algorithm can select smaller feature subsets with higher classification performance than state-of-the-art evolutionary computation (EC) and non-EC benchmark algorithms.

References

[1]

J. Liet al., “Feature selection: A data perspective,” ACM Comput. Surveys, vol. 50, no. 6, pp. 1–45, 2018.

Digital Library

[2]

B. H. Nguyen, B. Xue, and M. Zhang, “A survey on swarm intelligence approaches to feature selection in data mining,” Swarm Evol. Comput., vol. 54, May 2020, Art. no.

[3]

W. Zheng, S. Chen, Z. Fu, F. Zhu, H. Yan, and J. Yang, “Feature selection boosted by unselected features,” IEEE Trans. Neural Netw. Learn. Syst., early access, May 1, 2021. 10.1109/TNNLS.2021.3058172.

[4]

K. Wang, R. He, L. Wang, W. Wang, and T. Tan, “Joint feature selection and subspace learning for cross-modal retrieval,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 10, pp. 2010–2023, Oct. 2016.

Digital Library

[5]

B. Xue, M. Zhang, W. N. Browne, and X. Yao, “A survey on evolutionary computation approaches to feature selection,” IEEE Trans. Evol. Comput., vol. 20, no. 4, pp. 606–626, Aug. 2016.

Digital Library

[6]

Z. Wang, F. Nie, L. Tian, R. Wang, and X. Li, “Discriminative feature selection via a structured sparse subspace learning module,” in Proc. 29th Int. Joint Conf. Artif. Intell. (IJCAI), 2020, pp. 3009–3015.

[7]

S. Gu, R. Cheng, and Y. Jin, “Feature selection for high-dimensional classification using a competitive swarm optimizer,” Soft Comput., vol. 22, no. 3, pp. 811–822, 2018.

[8]

X.-F. Song, Y. Zhang, D.-W. Gong, and X.-Y. Sun, “Feature selection using bare-bones particle swarm optimization with mutual information,” Pattern Recognit., vol. 112, Apr. 2021, Art. no.

[9]

X.-F. Song, Y. Zhang, Y.-N. Guo, X.-Y. Sun, and Y.-L. Wang, “Variable-size cooperative coevolutionary particle swarm optimization for feature selection on high-dimensional data,” IEEE Trans. Evol. Comput., vol. 24, no. 5, pp. 882–895, Oct. 2020.

[10]

Y. Hu, Y. Zhang, and D. Gong, “Multiobjective particle swarm optimization for feature selection with fuzzy cost,” IEEE Trans. Cybern., vol. 51, no. 2, pp. 874–888, Feb. 2021.

[11]

B. H. Nguyen, B. Xue, P. Andreae, and M. Zhang, “A new binary particle swarm optimization approach: Momentum and dynamic balance between exploration and exploitation,” IEEE Trans. Cybern., vol. 51, no. 2, pp. 589–603, Feb. 2021.

[12]

R. Cheng and Y. Jin, “A competitive swarm optimizer for large scale optimization,” IEEE Trans. Cybern., vol. 45, no. 2, pp. 191–204, Feb. 2015.

[13]

K. Mistry, L. Zhang, S. C. Neoh, C. P. Lim, and B. Fielding, “A micro-GA embedded PSO feature selection approach to intelligent facial emotion recognition,” IEEE Trans. Cybern., vol. 47, no. 6, pp. 1496–1509, Jun. 2017.

[14]

B. Tran, B. Xue, and M. Zhang, “Variable-length particle swarm optimization for feature selection on high-dimensional classification,” IEEE Trans. Evol. Comput., vol. 23, no. 3, pp. 473–487, Jun. 2019.

[15]

H. B. Nguyen, B. Xue, I. Liu, P. Andreae, and M. Zhang, “New mechanism for archive maintenance in PSO-based multi-objective feature selection,” Soft Comput., vol. 20, no. 10, pp. 3927–3946, 2016.

Digital Library

[16]

R. J. Urbanowicz, M. Meeker, W. La Cava, R. S. Olson, and J. H. Moore, “Relief-based feature selection: Introduction and review,” J. Biomed. Inform., vol. 85, pp. 189–203, Sep. 2018.

Digital Library

[17]

M. A. Hall, “Correlation-based feature selection of discrete and numeric class machine learning,” Univ. Waikato, Hamilton, New Zealand, Working Paper 00/08, 2000.

[18]

H. Peng, F. Long, and C. Ding, “Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 8, pp. 1226–1238, Aug. 2005.

Digital Library

[19]

Z. Zhao, L. Wang, H. Liu, and J. Ye, “On similarity preserving feature selection,” IEEE Trans. Knowl. Data Eng., vol. 25, no. 3, pp. 619–632, Mar. 2013.

Digital Library

[20]

J. Gui, Z. Sun, S. Ji, D. Tao, and T. Tan, “Feature selection based on structured sparsity: A comprehensive study,” IEEE Trans. Neural Netw. Learn. Syst., vol. 28, no. 7, pp. 1490–1507, Jul. 2017.

[21]

F. Nie, H. Huang, X. Cai, and C. Ding, “Efficient and robust feature selection via joint ł_2,1-norms minimization,” in Proc. Int. Conf. Adv. Neural Inf. Process. Syst., vol. 23, 2010, pp. 1813–1821.

[22]

H. Peng and Y. Fan, “A general framework for sparsity regularized feature selection via iteratively reweighted least square minimization,” in Proc. 31st AAAI Conf. Artif. Intell., vol. 31, 2017, pp. 2471–2477.

[23]

M. Fan, X. Chang, X. Zhang, D. Wang, and L. Du, “Top-k supervise feature selection via ADMM for integer programming,” in Proc. 26th Int. Joint Conf. Artif. Intell. (IJCAI), 2017, pp. 1646–1653.

[24]

I.-S. Oh, J.-S. Lee, and B.-R. Moon, “Hybrid genetic algorithms for feature selection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 11, pp. 1424–1437, Nov. 2004.

Digital Library

[25]

C. Freeman, D. Kulić, and O. Basir, “Feature-selected tree-based classification,” IEEE Trans. Cybern., vol. 43, no. 6, pp. 1990–2004, Dec. 2013.

[26]

Z. Tao, L. Huiling, W. Wenwen, and Y. Xia, “GA-SVM based feature selection and parameter optimization in hospitalization expense modeling,” Appl. Soft Comput., vol. 75, pp. 323–332, Feb. 2019.

[27]

Y. Zhu, J. Liang, J. Chen, and Z. Ming, “An improved NSGA-III algorithm for feature selection used in intrusion detection,” Knowl. Based Syst., vol. 116, pp. 74–85, Jan. 2017.

Digital Library

[28]

A.-D. Li, B. Xue, and M. Zhang, “Multi-objective feature selection using hybridization of a genetic algorithm and direct multisearch for key quality characteristic selection,” Inf. Sci., vol. 523, pp. 245–265, Jun. 2020.

[29]

P. Shunmugapriya and S. Kanmani, “A hybrid algorithm using ant and bee colony optimization for feature selection and classification (AC-ABC hybrid),” Swarm Evol. Comput., vol. 36, pp. 27–36, Oct. 2017.

[30]

E. Hancer, B. Xue, and M. Zhang, “Differential evolution for filter feature selection based on information theory and feature ranking,” Knowl. Based Syst., vol. 140, pp. 103–119, Jan. 2018.

Digital Library

[31]

Y. Zhang, D.-W. Gong, X.-Z. Gao, T. Tian, and X.-Y. Sun, “Binary differential evolution with self-learning for multi-objective feature selection,” Inf. Sci., vol. 507, pp. 67–85, Jan. 2020.

Digital Library

[32]

R. Rivera-López, E. Mezura-Montes, J. Canul-Reich, and M. A. Cruz-Chávez, “A permutational-based differential evolution algorithm for feature subset selection,” Pattern Recognit. Lett., vol. 133, pp. 86–93, May 2020.

[33]

S.-W. Lin, K.-C. Ying, S.-C. Chen, and Z.-J. Lee, “Particle swarm optimization for parameter determination and feature selection of support vector machines,” Expert Syst. Appl., vol. 35, no. 4, pp. 1817–1824, 2008.

Digital Library

[34]

H. B. Nguyen, B. Xue, I. Liu, and M. Zhang, “Filter based backward elimination in wrapper based PSO for feature selection in classification,” in Proc. IEEE Congr. Evol. Comput. (CEC), 2014, pp. 3111–3118.

[35]

M.-Y. Cho and T. T. Hoang, “Feature selection and parameters optimization of SVM using particle swarm optimization for fault classification in power distribution systems,” Comput. Intell. Neurosci., vol. 2017, Jul. 2017, Art. no.

[36]

B. Tran, B. Xue, and M. Zhang, “A new representation in PSO for discretization-based feature selection,” IEEE Trans. Cybern., vol. 48, no. 6, pp. 1733–1746, Jun. 2018.

[37]

B. Tran, M. Zhang, and B. Xue, “A PSO based hybrid feature selection algorithm for high-dimensional classification,” in Proc. IEEE Congr. Evol. Comput. (CEC), 2016, pp. 3801–3808.

[38]

B. Tran, B. Xue, and M. Zhang, “Improved PSO for feature selection on high-dimensional datasets,” in Proc. Asia–Pacific Conf. Simul. Evol. Learn., 2014, pp. 503–515.

[39]

H. B. Nguyen, B. Xue, P. Andreae, and M. Zhang, “Particle swarm optimisation with genetic operators for feature selection,” in Proc. IEEE Congr. Evol. Comput. (CEC), 2017, pp. 286–293.

[40]

B. H. Nguyen, B. Xue, P. Andreae, H. Ishibuchi, and M. Zhang, “Multiple reference points-based decomposition for multiobjective feature selection in classification: Static and dynamic mechanisms,” IEEE Trans. Evol. Comput., vol. 24, no. 1, pp. 170–184, Feb. 2020.

Digital Library

[41]

H. Ishibuchi, K. Nozaki, N. Yamamoto, and H. Tanaka, “Selecting fuzzy if-then rules for classification problems using genetic algorithms,” IEEE Trans. Fuzzy Syst., vol. 3, no. 3, pp. 260–270, Aug. 1995.

Digital Library

[42]

K. Chen, B. Xue, M. Zhang, and F. Zhou, “An evolutionary multitasking-based feature selection method for high-dimensional classification,” IEEE Trans. Cybern., vol. 52, no. 7, pp. 7172–7186, Jul. 2022.

[43]

X. Wuet al., “Top 10 algorithms in data mining,” Knowl. Inf. Syst., vol. 14, no. 1, pp. 1–37, 2008.

Digital Library

[44]

P. Mohapatra, K. N. Das, and S. Roy, “A modified competitive swarm optimizer for large scale optimization problems,” Appl. Soft Comput., vol. 59, pp. 340–362, Oct. 2017.

Digital Library

[45]

B. Xue, M. Zhang, and W. N. Browne, “A comprehensive comparison on evolutionary feature selection approaches to classification,” Int. J. Comput. Intell. Appl., vol. 14, no. 2, 2015, Art. no.

[46]

H. Xu, B. Xue, and M. Zhang, “A duplication analysis-based evolutionary algorithm for biobjective feature selection,” IEEE Trans. Evol. Comput., vol. 25, no. 2, pp. 205–218, Apr. 2021.

[47]

O. Olorunda and A. P. Engelbrecht, “Measuring exploration/exploitation in particle swarms using swarm diversity,” in Proc. IEEE Congr. Evol. Comput., 2008, pp. 1128–1134.

Cited By

Bu CLiu YHuang MShao JJi SLuo WWu X(2024)Layer-Wise Learning Rate Optimization for Task-Dependent Fine-Tuning of Pre-Trained Models: An Evolutionary ApproachACM Transactions on Evolutionary Learning and Optimization10.1145/36898274:4(1-23)Online publication date: 24-Aug-2024
https://dl.acm.org/doi/10.1145/3689827
Zhang YDu KJiang YWang LWang HZhan ZLi XHandl J(2024)Adaptive Aggregative Multitask Competitive Particle Swarm Optimization with Bi-Directional Asymmetric Flip Strategy for High-Dimensional Feature SelectionProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654021(1515-1523)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638529.3654021
Jin XWei BDeng LYang SZheng JWang F(2024)An adaptive pyramid PSO for high-dimensional feature selectionExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.125084257:COnline publication date: 10-Dec-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.125084

Index Terms

A Constrained Competitive Swarm Optimizer With an SVM-Based Surrogate Model for Feature Selection

Index terms have been assigned to the content through auto-classification.

Recommendations

A multi-strategy surrogate-assisted competitive swarm optimizer for expensive optimization problems
Abstract
Evolutionary computation is a powerful tool for solving nonconvex optimization problems. Generally, evolutionary algorithms take numerous fitness evaluations to obtain the potential optimal solutions. This poses a critical challenge for applying ...
Highlights
- SACSO is proposed to solve expensive optimization problems and applied to speed reducer design.
- Particle selection criteria contain global search, local search, and opposition-based search.
- A dynamic adaptation strategy is proposed ...
A modified competitive swarm optimizer for large scale optimization problems

Display Omitted The proposed work (MCSO) is motivated by the Competitive Swarm Optimizer (CSO).2/3rd of the swarm are updated in MCSO every time by a tri-competitive criteria.Both CEC 2008 and CEC 2010 benchmark functions have been solved using ...
A Surrogate-Assisted Ensemble Particle Swarm Optimizer for Feature Selection Problems
Advances in Swarm Intelligence
Abstract
For feature selection problems on high-dimensional data, this paper proposes a surrogate-assisted ensemble particle swarm feature selection algorithm, by combining the global search ability of evolutionary algorithm with the fast search ability of ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Evolutionary Computation

IEEE Transactions on Evolutionary Computation Volume 28, Issue 1

Feb. 2024

279 pages

Issue’s Table of Contents

1089-778X © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 February 2024

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bu CLiu YHuang MShao JJi SLuo WWu X(2024)Layer-Wise Learning Rate Optimization for Task-Dependent Fine-Tuning of Pre-Trained Models: An Evolutionary ApproachACM Transactions on Evolutionary Learning and Optimization10.1145/36898274:4(1-23)Online publication date: 24-Aug-2024
https://dl.acm.org/doi/10.1145/3689827
Zhang YDu KJiang YWang LWang HZhan ZLi XHandl J(2024)Adaptive Aggregative Multitask Competitive Particle Swarm Optimization with Bi-Directional Asymmetric Flip Strategy for High-Dimensional Feature SelectionProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654021(1515-1523)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638529.3654021
Jin XWei BDeng LYang SZheng JWang F(2024)An adaptive pyramid PSO for high-dimensional feature selectionExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.125084257:COnline publication date: 10-Dec-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.125084

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents