Distributed Active Client Selection With Noisy Clients Using Model Association Scores
Pages 75 - 92
Abstract
Active client selection (ACS) strategically identifies clients for model updates during each training round of federated learning. In scenarios with limited communication resources, ACS emerges as a superior alternative to random client selection, significantly improving the convergence rate. However, existing ACS methods struggle with clients providing noisy updates, e.g. those from noisy labels. To address this challenge, we present a new ACS algorithm for scenarios with unknown noisy clients. Our algorithm constructs a client sampling distribution based on the global association among model updates, which quantifies the ability of a client’s model update to align with those from other clients. By leveraging these associations, we efficiently identify and mitigate the impact of clients with substantial noise that could disrupt training. This approach is simple, computationally efficient, and requires no hyperparameter tuning. Experiments on six benchmark datasets demonstrate that conventional ACS methods fail to outperform random selection. In contrast, our approach significantly enhances convergence speed while using the same communication resources.
References
[1]
Allouah, Y., Farhadkhani, S., Guerraoui, R., Gupta, N., Pinot, R., Stephan, J.: Fixing by mixing: a recipe for optimal Byzantine ML under heterogeneity. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (2023)
[2]
Amiri, M.M., Gunduz, D., Kulkarni, S.R., Poor, H.V.: Federated learning with quantized global model updates. arXiv arXiv:2006.10672 (2020)
[3]
Bell, S., Upchurch, P., Snavely, N., Bala, K.: Material recognition in the wild with the materials in context database. In: CVPR, pp. 3479–3487 (2015)
[4]
Bengar, J.Z., van de Weijer, J., Twardowski, B., Raducanu, B.: Reducing label effort: self-supervised meets active learning. In: ICCV Workshops (2021)
[5]
Bernhardt, M., et al.: Active label cleaning for improved dataset quality under resource constraints. Nat. Commun. 13 (2022)
[6]
Biswas, S., Young, K., Griffith, J.: A comparison of automatic labelling approaches for sentiment analysis. arXiv arXiv:2211.02976 (2022)
[7]
Blanchard, P., Mhamdi, E.M.E., Guerraoui, R., Stainer, J.: Machine learning with adversaries: Byzantine tolerant gradient descent. In: NIPS (2017)
[8]
Bossard L, Guillaumin M, and Van Gool L Fleet D, Pajdla T, Schiele B, and Tuytelaars T Food-101 – mining discriminative components with random forests Computer Vision – ECCV 2014 2014 Cham Springer 446-461
[9]
Cho, Y.J., Wang, J., Joshi, G.: Towards understanding biased client selection in federated learning. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (2022)
[10]
Cohen, G., Afshar, S., Tapson, J., van Schaik, A.: EMNIST: extending MNIST to handwritten letters. In: International Joint Conference on Neural Networks, pp. 2921–2926 (2017)
[11]
Dai, R., Shen, L., He, F., Tian, X., Tao, D.: DisPFL: towards communication-efficient personalized federated learning via decentralized sparse training. In: Proceedings of the International Conference on Machine Learning, pp. 162:4587–162:4604 (2022)
[12]
Darlow, L.N., Crowley, E.J., Antoniou, A., Storkey, A.J.: CINIC-10 is not ImageNet or CIFAR-10. arXiv arXiv:1810.03505 (2018)
[13]
Data, D., Diggavi, S.: Byzantine-resilient high-dimensional SGD with local iterations on heterogeneous data. In: Proceedings of the International Conference on Machine Learning (2021)
[14]
Fraboni, Y., Vidal, R., Kameni, L., Lorenzi, M.: Clustered sampling: low-variance and improved representativity for clients selection in federated learning. In: Proceedings of the International Conference on Machine Learning, pp. 139:3407–139:3416 (2021)
[15]
Goetz, J., Malik, K., Bui, D., Moon, S., Liu, H., Kumar, A.: Active federated learning. arXiv arXiv:1909.12641 (2019)
[16]
Han, B., et al.: Masking: a new perspective of noisy supervision. In: NeurIPS (2018)
[17]
Han, B., et al.: Co-teaching: robust training of deep neural networks with extremely noisy labels. In: NeurIPS (2018)
[18]
Hsu T-MH, Qi H, and Brown M Vedaldi A, Bischof H, Brox T, and Frahm J-M Federated visual classification with real-world data distribution Computer Vision – ECCV 2020 2020 Cham Springer 76-92
[19]
Huang, H., et al.: Active client selection for clustered federated learning. IEEE Trans. Neural Netw. Learn. Syst. (2023, early access)
[20]
Huang, J., Qu, L., Jia, R., Zhao, B.: O2U-Net: a simple noisy label detection approach for deep neural networks. In: Proceedings of the International Conference on Computer Vision, pp. 3326–3334 (2019)
[21]
Jiang, L., Zhou, Z., Leung, T., Li, L.J., Fei-Fei, L.: MentorNet: learning data-driven curriculum for very deep neural networks on corrupted labels. In: Proceedings of the International Conference on Machine Learning (2018)
[22]
Karimireddy, S.P., He, L., Jaggi, M.: Byzantine-robust learning on heterogeneous datasets via bucketing. In: Proceedings of the International Conference on Learning Representations (2022)
[23]
Kim, T., Bae, S., woo Lee, J., Yun, S.: Accurate and fast federated learning via combinatorial multi-armed bandits. arXiv arXiv:2012.03270 (2020)
[24]
Kremer, J., Sha, F., Igel, C.: Robust active label correction. In: Proceedings of the International Conference on Artificial Intelligence and Statistics, pp. 308–316 (2018)
[25]
Krizhevsky, A.: Learning multiple layers of features from tiny images. University of Toronto, Technical report (2009)
[26]
Lai, F., Zhu, X., Madhyastha, H.V., Chowdhury, M.: Oort: efficient federated learning via guided participant selection. In: Proceedings of the USENIX Symposium on Operating Systems Design and Implementation, pp. 162:4587–162:4604 (2021)
[27]
Li, L., Xu, W., Chen, T., Giannakis, G.B., Ling, Q.: RSA: Byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets. In: AAAI (2019)
[28]
Li, P., Zhao, Y., Chen, L., Cheng, K., Xie, C., Wang, X., Hu, Q.: Uncertainty measured active client selection for federated learning in smart grid. In: Proceedings of the International Conference on Smart Internet of Things, pp. 148–153 (2022)
[29]
Liu, Y., Chen, C., Lyu, L., Wu, F., Wu, S., Chen, G.: Byzantine-robust learning on heterogeneous data via gradient splitting. In: Proceedings of the International Conference on Machine Learning (2023)
[30]
McMahan, H.B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (2017)
[31]
Nam, H.W., Moon, Y.B., Oh, T.H.: FedPara: low-rank Hadamard product for communication-efficient federated learning. In: Proceedings of the International Conference on Learning Representations (2022)
[32]
Northcutt, C.G., Athalye, A., Mueller, J.: Pervasive label errors in test sets destabilize machine learning benchmarks. In: NeurIPS (2021)
[33]
Park, S., Jo, D.U., Choi, J.Y.: Over-fit: noisy-label detection based on the overfitted model property. arXiv arXiv:2106.07217 (2021)
[34]
Pillutla K, Kakade SM, and Harchaoui Z Robust aggregation for federated learning IEEE Trans. Sig. Process. 2022 70 1142-1154
[35]
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., Pedarsani, R.: FedPAQ: a communication-efficient federated learning method with periodic averaging and quantization. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (2020)
[36]
Ribero, M., Vikalo, H.: Communication-efficient federated learning via optimal client sampling. arXiv arXiv:2007.15197 (2020)
[37]
Tang, M., et al.: FedCor: correlation-based active client selection strategy for heterogeneous federated learning. In: CVPR, pp. 10102–10111 (2022)
[38]
Turan B, Uribe CA, Wai HT, and Alizadeh M Robust distributed optimization with randomly corrupted gradients IEEE Trans. Sig. Process. 2022 70 3484-3498
[39]
Wang, H., Liu, B., Li, C., Yang, Y., Li, T.: Learning with noisy labels for sentence-level sentiment classification. In: EMNLP-IJCNLP (2019)
[40]
Wang, Y., Ma, X., Chen, Z., Luo, Y., Yi, J.F., Bailey, J.: Symmetric cross entropy for robust learning with noisy labels. In: Proceedings of the International Conference on Computer Vision (2019)
[41]
Yu, X., Han, B., Yao, J., Niu, G., Tsang, I.W., Sugiyama, M.: How does disagreement help generalization against label corruption? In: Proceedings of the International Conference on Machine Learning (2019)
[42]
Zhang, Z., Sabuncu, M.R.: Generalized cross entropy loss for training deep neural networks with noisy labels. In: NeurIPS (2018)
[43]
Zhu, Z., Hong, J., Drew, S., Zhou, J.: Resilient and communication efficient learning for heterogeneous federated systems. In: Proceedings of the International Conference on Machine Learning, pp. 162:27504–162:27526 (2022)
Index Terms
- Distributed Active Client Selection With Noisy Clients Using Model Association Scores
Index terms have been assigned to the content through auto-classification.
Recommendations
Towards an Efficient Client Selection System for Federated Learning
Cloud Computing – CLOUD 2022AbstractFederated learning is a popular distributed machine learning model where a centralized server orchestrates many distributed clients to coordinate the completion of model training or evaluation without sharing private or local data. More and more ...
Tackling Noisy Clients in Federated Learning with End-to-end Label Correction
CIKM '24: Proceedings of the 33rd ACM International Conference on Information and Knowledge ManagementRecently, federated learning (FL) has achieved wide successes for diverse privacy-sensitive applications without sacrificing the sensitive private information of clients. However, the data quality of client datasets can not be guaranteed since ...
Noisy multi-label semi-supervised dimensionality reduction
Highlights- A new semi-supervised and label noise-tolerant multi-label dimensionality reduction method.
AbstractNoisy labeled data represent a rich source of information that often are easily accessible and cheap to obtain, but label noise might also have many negative consequences if not accounted for. How to fully utilize noisy labels has been ...
Comments
Information & Contributors
Information
Published In
Sep 2024
571 pages
ISBN:978-3-031-73026-9
DOI:10.1007/978-3-031-73027-6
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 26 November 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 09 Jan 2025