Article

Evasion attacks against machine learning at test time

Authors:

Battista Biggio,

Davide Maiorca,

Nedim Šrndić,

Giorgio Giacinto,

Fabio RoliAuthors Info & Claims

ECMLPKDD'13: Proceedings of the 2013th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part III

Pages 387 - 402

https://doi.org/10.1007/978-3-642-40994-3_25

Published: 23 September 2013 Publication History

Abstract

In security-sensitive applications, the success of machine learning depends on a thorough vetting of their resistance to adversarial data. In one pertinent, well-motivated attack scenario, an adversary may attempt to evade a deployed system at test time by carefully manipulating attack samples. In this work, we present a simple but effective gradient-based approach that can be exploited to systematically assess the security of several, widely-used classification algorithms against evasion attacks. Following a recently proposed framework for security evaluation, we simulate attack scenarios that exhibit different risk levels for the classifier by increasing the attacker's knowledge of the system and her ability to manipulate attack samples. This gives the classifier designer a better picture of the classifier performance under evasion attacks, and allows him to perform a more informed model selection (or parameter setting). We evaluate our approach on the relevant security task of malware detection in PDF files, and show that such systems can be easily evaded. We also sketch some countermeasures suggested by our analysis.

References

[1]

Adobe: PDF Reference, sixth edn. version 1.7

[2]

Barreno, M., Nelson, B., Sears, R., Joseph, A.D., Tygar, J.D.: Can machine learning be secure? In: ASIACCS 2006: Proc. of the 2006 ACM Symp. on Information, Computer and Comm. Security, pp. 16-25. ACM, New York (2006)

[3]

Biggio, B., Fumera, G., Roli, F.: Multiple classifier systems for robust classifier design in adversarial environments. Int'l J. of Machine Learning and Cybernetics 1(1), 27-41 (2010)

[4]

Biggio, B., Fumera, G., Roli, F.: Design of robust classifiers for adversarial environments. In: IEEE Int'l Conf. on Systems, Man, and Cybernetics (SMC), pp. 977-982 (2011)

[5]

Biggio, B., Fumera, G., Roli, F.: Security evaluation of pattern classifiers under attack. IEEE Trans. on Knowl. and Data Eng. 99(PrePrints), 1 (2013)

[6]

Biggio, B., Nelson, B., Laskov, P.: Poisoning attacks against support vector machines. In: Langford, J., Pineau, J. (eds.) 29th Int'l Conf. on Mach. Learn. (2012)

[7]

Brückner, M., Scheffer, T.: Stackelberg games for adversarial prediction problems. In: Knowl. Disc. and D. Mining (KDD), pp. 547-555 (2011)

[8]

Brückner, M., Kanzow, C., Scheffer, T.: Static prediction games for adversarial learning problems. J. Mach. Learn. Res. 13, 2617-2654 (2012)

[9]

Dalvi, N., Domingos, P., Mausam, S.S., Verma, D.: Adversarial classification. In: 10th ACM SIGKDD Int'l Conf. on Knowl. Discovery and Data Mining (KDD), pp. 99-108 (2004)

[10]

Dekel, O., Shamir, O., Xiao, L.: Learning to classify with missing and corrupted features. Mach. Learn. 81, 149-178 (2010)

[11]

Fogla, P., Sharif, M., Perdisci, R., Kolesnikov, O., Lee, W.: Polymorphic blending attacks. In: Proc. 15th Conf. on USENIX Sec. Symp. USENIX Association, CA (2006)

[12]

Globerson, A., Roweis, S.T.: Nightmare at test time: robust learning by feature deletion. In: Cohen, W.W., Moore, A. (eds.) Proc. of the 23rd Int'l Conf. on Mach. Learn., vol. 148, pp. 353-360. ACM (2006)

[13]

Golland, P.: Discriminative direction for kernel classifiers. In: Neu. Inf. Proc. Syst (NIPS), pp. 745-752 (2002)

[14]

Huang, L., Joseph, A.D., Nelson, B., Rubinstein, B., Tygar, J.D.: Adversarial machine learning. In: 4th ACM Workshop on Art. Int. and Sec (AISec 2011), Chicago, IL, USA, pp. 43-57 (2011)

[15]

Kloft, M., Laskov, P.: Online anomaly detection under adversarial impact. In: Proc. of the 13th Int'l Conf. on Art. Int. and Stats (AISTATS), pp. 405-412 (2010)

[16]

Kolcz, A., Teo, C.H.: Feature weighting for improved classifier robustness. In: Sixth Conf. on Email and Anti-Spam (CEAS), Mountain View, CA, USA (2009)

[17]

Laskov, P., Kloft, M.: A framework for quantitative security analysis of machine learning. In: AISec 2009: Proc. of the 2nd ACM Works. on Sec. and Art. Int., pp. 1-4. ACM, New York (2009)

[18]

LeCun, Y., Jackel, L., Bottou, L., Brunot, A., Cortes, C., Denker, J., Drucker, H., Guyon, I., Müller, U., Säckinger, E., Simard, P., Vapnik, V.: Comparison of learning algorithms for handwritten digit recognition. In: Int'l Conf. on Art. Neu. Net., pp. 53-60 (1995)

[19]

Lowd, D., Meek, C.: Adversarial learning. In: Press, A. (ed.) Proc. of the Eleventh ACM SIGKDD Int'l Conf. on Knowl. Disc. and D. Mining (KDD), Chicago, IL, pp. 641-647 (2005)

[20]

Maiorca, D., Giacinto, G., Corona, I.: A pattern recognition system for malicious pdf files detection. In: MLDM, pp. 510-524 (2012)

[21]

Nelson, B., Barreno, M., Chi, F.J., Joseph, A.D., Rubinstein, B.I.P., Saini, U., Sutton, C., Tygar, J.D., Xia, K.: Exploiting machine learning to subvert your spam filter. In: LEET 2008: Proc. of the 1st USENIX Work. on L.-S. Exp. and Emerg. Threats, pp. 1-9. USENIX Association, Berkeley (2008)

[22]

Nelson, B., Rubinstein, B.I., Huang, L., Joseph, A.D., Lee, S.J., Rao, S., Tygar, J.D.: Query strategies for evading convex-inducing classifiers. J. Mach. Learn. Res. 13, 1293-1332 (2012)

[23]

Platt, J.: Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In: Smola, A., Bartlett, P., Schölkopf, B., Schuurmans, D. (eds.) Adv. in L. M. Class, pp. 61-74 (2000)

[24]

Smutz, C., Stavrou, A.: Malicious pdf detection using metadata and structural features. In: Proc. of the 28th Annual Comp. Sec. App. Conf., pp. 239-248 (2012)

[25]

Šrndić, N., Laskov, P.: Detection of malicious pdf files based on hierarchical document structure. In: Proc. 20th Annual Net. & Dist. Sys. Sec. Symp. (2013)

[26]

Young, R.: 2010 IBM X-force mid-year trend & risk report. Tech. rep., IBM (2010)

Cited By

Xiang CWu TDai SPetit JJana SMittal PBalzarotti DXu W(2024)PATCHCUREProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699106(3675-3692)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699106
Dahiya PShumailov IAnderson RBalzarotti DXu W(2024)Machine learning needs better randomness standardsProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699105(3657-3674)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699105
Wang ZZhu RZhou DZhang ZMitchell JTang HWang XBalzarotti DXu W(2024)DPAdapterProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698956(991-1008)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3698956
Show More Cited By

Evasion attacks against machine learning at test time
1. Computing methodologies
2. Social and professional topics
  1. Computing / technology policy

Recommendations

Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain

In recent years, machine learning algorithms, and more specifically deep learning algorithms, have been widely used in many fields, including cyber security. However, machine learning systems are vulnerable to adversarial attacks, and this limits the ...
Using Honeypots in a Decentralized Framework to Defend Against Adversarial Machine-Learning Attacks
Applied Cryptography and Network Security Workshops
Abstract
The market demand for online machine-learning services is increasing, and so have the threats against them. Adversarial inputs represent a new threat to Machine-Learning-as-a-Services (MLaaSs). Meticulously crafted malicious inputs can be used to ...
Machine Learning under Attack: Vulnerability Exploitation and Security Measures
IH&MMSec '16: Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security

Learning to discriminate between secure and hostile patterns is a crucial problem for species to survive in nature. Mimetism and camouflage are well-known examples of evolving weapons and defenses in the arms race between predators and preys. It is thus ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

ECMLPKDD'13: Proceedings of the 2013th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part III

September 2013

685 pages

ISBN:9783642409936

Editors:
Hendrik Blockeel
Department of Computer Science, Katholieke Universiteit Leuven, Leuven, Belgium
,
Kristian Kersting
Fraunhofer IAIS, Department of Knowledge Discovery, Schloss Birlinghoven, University of Bonn, Sankt Augustin, Germany
,
Siegfried Nijssen
LIACS, Universiteit Leiden, Leiden, The Netherlands and KU Leuven, Department of Computer Science, Leuven, Belgium
,
Filip Železný
Department of Computer Science and Engineering, Czech Technical University, Prague 6, Czech Republic

Sponsors

XRCE: Xerox Research Centre Europe
Winton Capital Management: Winton Capital Management
Cisco Systems
Yahoo! Labs
CSKI: Czech Society for Cybernetics and Informatics

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 23 September 2013

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

278
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xiang CWu TDai SPetit JJana SMittal PBalzarotti DXu W(2024)PATCHCUREProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699106(3675-3692)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699106
Dahiya PShumailov IAnderson RBalzarotti DXu W(2024)Machine learning needs better randomness standardsProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699105(3657-3674)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699105
Wang ZZhu RZhou DZhang ZMitchell JTang HWang XBalzarotti DXu W(2024)DPAdapterProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698956(991-1008)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3698956
Cullen ALiu SMontague PErfani SRubinstein BSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Et Tu certificationsProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692455(9745-9761)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3692455
Kamath SMittal SDeshpande ABalasubramanian VWooldridge MDy JNatarajan S(2024)Rethinking robustness of model attributionsProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i3.28047(2688-2696)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i3.28047
Cen SAlur R(2024)From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI AuditingProceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization10.1145/3689904.3694711(1-14)Online publication date: 29-Oct-2024
https://dl.acm.org/doi/10.1145/3689904.3694711
Lucas KLin WBauer LReiter MSharif MLuo BLiao XXu JKirda ELie D(2024)Training Robust ML-based Raw-Binary Malware Detectors in Hours, not MonthsProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security10.1145/3658644.3690208(124-138)Online publication date: 2-Dec-2024
https://dl.acm.org/doi/10.1145/3658644.3690208
Carletti VFoggia PGreco ASaggese AVento M(2024)Facial Soft-biometrics Obfuscation through Adversarial AttacksACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365647420:11(1-21)Online publication date: 12-Sep-2024
https://dl.acm.org/doi/10.1145/3656474
Ge YLiu SFu ZTan JLi ZXu SLi YXian YZhang Y(2024)A Survey on Trustworthy Recommender SystemsACM Transactions on Recommender Systems10.1145/36528913:2(1-68)Online publication date: 13-Apr-2024
https://dl.acm.org/doi/10.1145/3652891
Arp DQuiring EPendlebury FWarnecke APierazzi FWressnegger CCavallaro LRieck K(2024)Pitfalls in Machine Learning for Computer SecurityCommunications of the ACM10.1145/364345667:11(104-112)Online publication date: 25-Oct-2024
https://dl.acm.org/doi/10.1145/3643456
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents