Article

Totally corrective boosting algorithms that maximize the margin

Authors:

Manfred K. Warmuth,

Gunnar RätschAuthors Info & Claims

ICML '06: Proceedings of the 23rd international conference on Machine learning

Pages 1001 - 1008

https://doi.org/10.1145/1143844.1143970

Published: 25 June 2006 Publication History

Abstract

We consider boosting algorithms that maintain a distribution over a set of examples. At each iteration a weak hypothesis is received and the distribution is updated. We motivate these updates as minimizing the relative entropy subject to linear constraints. For example AdaBoost constrains the edge of the last hypothesis w.r.t. the updated distribution to be at most γ = 0. In some sense, AdaBoost is "corrective" w.r.t. the last hypothesis. A cleaner boosting method is to be "totally corrective": the edges of all past hypotheses are constrained to be at most γ, where γ is suitably adapted.Using new techniques, we prove the same iteration bounds for the totally corrective algorithms as for their corrective versions. Moreover with adaptive γ, the algorithms provably maximizes the margin. Experimentally, the totally corrective versions return smaller convex combinations of weak hypotheses than the corrective ones and are competitive with LPBoost, a totally corrective boosting algorithm with no regularization, for which there is no iteration bound known.

References

[1]

Bennett, K., Demiriz, A., & Shawe-Taylor, J. (2000). A column generation algorithm for boosting. Proc. ICML (pp. 65--72). Morgan Kaufmann.

Digital Library

[2]

Bregman, L. (1967). The relaxation method for finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Computational Math. and Math. Physics, 7, 200--127.

[3]

Breiman, L. (1997). Prediction games and arcing algorithmsTechnical Report 504). Statistics Department, University of California at Berkeley.

[4]

Breiman, L. (1999). Prediction games and arcing algorithms. Neural Computation, 11, 1493--1518.

Digital Library

[5]

Duffy, N., & Helmbold, D. (2000). Potential boosters? NIPS'00 (pp. 258--264).

[6]

Freund, Y., & Schapire, R. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. J. of Comp. & Sys. Sci., 55, 119--139.

Digital Library

[7]

Friedman, J., Hastie, T., & Tibshirani, R. (2000). Additive Logistic Regression: a statistical view of boosting. Annals of Statistics, 2, 337--374.

[8]

Grove, A., & Schuurmans, D. (1998). Boosting in the limit: Maximizing the margin of learned ensembles. Proc. 15th Nat. Conf. on Art. Int.

Digital Library

[9]

Herbster, M., & Warmuth, M. (2001). Tracking the best linear prediction. J. Mach. Learn. Res., 281--309.

Digital Library

[10]

Kivinen, J., & Warmuth, M. (1999). Boosting as entropy projection. COLT'99.

Digital Library

[11]

Lafferty, J. (1999). Additive models, boosting, and inference for generalized divergences. COLT'99 (pp. 125--133).

Digital Library

[12]

Littlestone, N. (1988). Learning when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2, 285--318.

Digital Library

[13]

Long, P. M., & Wu, X. (2005). Mistake bounds for maximum entropy discrimination. NIPS'04 (pp. 833--840).

[14]

Nocedal, J., & Wright, S. (2000). Numerical optimization. Springer Series in Op. Res. Springer.

[15]

Rätsch, G., Onoda, T., & Müüller, K.-R. (2001). Soft margins for AdaBoost. Machine Learning, 42, 287--320.

Digital Library

[16]

Rätsch, G., & Warmuth, M. K. (2005). Efficient margin maximizing with boosting. J. Mach. Learn. Res., 2131--2152.

Digital Library

[17]

Rudin, C., Daubechies, I., & Schapire, R. (2004a). Dynamics of AdaBoost: Cyclic behavior and convergence of margins. J. Mach. Learn. Res., 1557--1595.

Digital Library

[18]

Rudin, C., Schapire, R., & Daubechies, I. (2004b). Analysis of boosting algoritms using the smooth margin function: A study of three algorithms. Unpublished manuscript.

[19]

Schapire, R., Freund, Y., Bartlett, P., & Lee, W. (1998). Boosting the margin: A new explanation for the effectiveness of voting methods. The Annals of Statistics, 26, 1651--1686.

[20]

Schapire, R., & Singer, Y. (1999). Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37, 297--336.

Digital Library

Cited By

Wang XWang YMa ZWong KLi X(2024)Exhaustive Exploitation of Nature-Inspired Computation for Cancer Screening in an Ensemble MannerIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2024.338540221:5(1366-1379)Online publication date: Sep-2024
https://doi.org/10.1109/TCBB.2024.3385402
Guo HLi HCong LWang W(2024)Online concept evolution detection based on active learningData Mining and Knowledge Discovery10.1007/s10618-024-01011-438:4(1589-1633)Online publication date: 15-Mar-2024
https://doi.org/10.1007/s10618-024-01011-4
Gupta YZhai RSuggala ARavikumar POh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Responsible AI (RAI) games and ensemblesProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669301(72717-72749)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3669301
Show More Cited By

Index Terms

Totally corrective boosting algorithms that maximize the margin
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Factorization methods
        Canonical correlation analysis
2. Mathematics of computing
  1. Probability and statistics
    1. Distribution functions
    2. Statistical paradigms
      1. Regression analysis
      2. Statistical graphics

Recommendations

Totally corrective boosting algorithms that maximize the margin
A direct formulation for totally-corrective multi-class boosting
CVPR '11: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition

Boosting combines a set of moderately accurate weak classifiers to form a highly accurate predictor. Compared with binary boosting classification, multi-class boosting received less attention. We propose a novel multi-class boosting formulation here. ...
Totally-corrective multi-class boosting
ACCV'10: Proceedings of the 10th Asian conference on Computer vision - Volume Part IV

We proffer totally-corrective multi-class boosting algorithms in this work. First, we discuss the methods that extend two-class boosting to multi-class case by studying two existing boosting algorithms: AdaBoost.MO and SAMME, and formulate convex ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '06: Proceedings of the 23rd international conference on Machine learning

June 2006

1154 pages

ISBN:1595933832

DOI:10.1145/1143844

Program Chairs:
William Cohen,
Andrew Moore

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Acceptance Rates

ICML '06 Paper Acceptance Rate 140 of 548 submissions, 26%;

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

62
Total Citations
View Citations
455
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang XWang YMa ZWong KLi X(2024)Exhaustive Exploitation of Nature-Inspired Computation for Cancer Screening in an Ensemble MannerIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2024.338540221:5(1366-1379)Online publication date: Sep-2024
https://doi.org/10.1109/TCBB.2024.3385402
Guo HLi HCong LWang W(2024)Online concept evolution detection based on active learningData Mining and Knowledge Discovery10.1007/s10618-024-01011-438:4(1589-1633)Online publication date: 15-Mar-2024
https://doi.org/10.1007/s10618-024-01011-4
Gupta YZhai RSuggala ARavikumar POh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Responsible AI (RAI) games and ensemblesProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669301(72717-72749)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3669301
Piotrowski DOrzeszko W(2023)Artificial intelligence and customers’ intention to use robo-advisory in banking servicesEquilibrium. Quarterly Journal of Economics and Economic Policy10.24136/eq.2023.03118:4(967-1007)Online publication date: 30-Dec-2023
https://doi.org/10.24136/eq.2023.031
Islam MXing L(2023)Cartography of Genomic Interactions Enables Deep Analysis of Single-Cell Expression DataNature Communications10.1038/s41467-023-36383-614:1Online publication date: 8-Feb-2023
https://doi.org/10.1038/s41467-023-36383-6
Guo HXia HLi HWang W(2023)Concept Evolution Detection Based on Noise Reduction Soft BoundaryInformation Sciences10.1016/j.ins.2023.01.115Online publication date: Feb-2023
https://doi.org/10.1016/j.ins.2023.01.115
Kim SKim J(2023)Sound-Based Abnormal Combustion Classification Model for High Compression Ratio, Spark-Ignition Engines Using Mel-Frequency Cepstrum Coefficients and Ensemble Learning AlgorithmsInternational Journal of Automotive Technology10.1007/s12239-023-0071-024:3(873-881)Online publication date: 12-May-2023
https://doi.org/10.1007/s12239-023-0071-0
Singh LSisodia DTaranath N(2023)Gradient Boosting-Based Predictive Click Fraud Detection Using Manifold Criterion Variable EliminationComputational Intelligence in Data Science10.1007/978-3-031-38296-3_22(287-299)Online publication date: 22-Jul-2023
https://doi.org/10.1007/978-3-031-38296-3_22
Gottlieb AYatsco ABakos-Block CLangabeer JChampagne-Langabeer T(2022)Machine Learning for Predicting Risk of Early Dropout in a Recovery Program for Opioid Use DisorderHealthcare10.3390/healthcare1002022310:2(223)Online publication date: 25-Jan-2022
https://doi.org/10.3390/healthcare10020223
Atif MFranzoni V(2022)Tell Me More: Automating Emojis Classification for Better Accessibility and Emotional Context RecognitionFuture Internet10.3390/fi1405014214:5(142)Online publication date: 5-May-2022
https://doi.org/10.3390/fi14050142
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents