Training Robust ML-based Raw-Binary Malware Detectors in Hours, not Months
Authors: 
Keane Lucas, 
Weiran Lin, 
Lujo Bauer, 
Michael K. Reiter, Mahmood SharifAuthors Info & Claims
Pages 124 - 138
Abstract
Machine-learning (ML) classifiers are increasingly used to distinguish malware from benign binaries. Recent work has shown that ML-based detectors can be evaded by adversarial examples, but also that one may defend against such attacks via adversarial training. However, adversarial training, and subsequent robustness evaluation, is computationally expensive in the raw-binary malware-detection domain because it requires producing many adversarial examples for both training and evaluation. Prior work found that Greedy-training, a faster robust training technique that forgoes using adversarial examples, showed some promise in producing robust malware detectors. However, Greedy-training was far less effective in inducing robustness than the more expensive adversarial training, and it also severely hurt natural accuracy (i.e., accuracy on the original data). To faster train models, this work presents GreedyBlock-training, an enhanced version of Greedy-training that we empirically show achieves not only state-of-the-art robustness in malware detectors, exceeding even adversarial training, but also retains natural accuracy better than adversarial training. Furthermore, as it does not require creating adversarial (or functional) examples, GreedyBlock-training is significantly faster than adversarial training. Specifically, we show that GreedyBlock-training can produce more robust (+54% on average), more naturally accurate (+7% on average), and more efficiently trained (-91% average computation) malware detectors than prior work. To faster evaluate models, we also develop methods to faster gauge the robustness of ML-based raw-binary malware detectors by introducing robustness proxies, which can be used either to predict which models are likely to be the most robust, thus helping prioritize which detectors to evaluate with expensive attacks, or aiding in deciding which detectors are worthwhile to continue training. Experimentally, we show these proxy measures can find the most robust detector in a pool of detectors while using only ~20-50% of the computation that would otherwise be required.
References
[1]
D. W. Aha and R. L. Bankert. A comparative evaluation of sequential feature selection algorithms. In Proc. AISTATS, 1995.
[2]
H. S. Anderson, A. Kharkar, B. Filar, and P. Roth. Evading machine learning malware detection. Black Hat, 2017.
[3]
H. S. Anderson and P. Roth. Ember: An open dataset for training static PE malware machine learning models. arXiv preprint, 2018.
[4]
D. Arp, M. Spreitzenbarth, M. Hubner, H. Gascon, K. Rieck, and C. Siemens. Drebin: Effective and explainable detection of android malware in your pocket. In Proc. NDSS, 2014.
[5]
S. Baluja and I. Fischer. Adversarial transformation networks: Learning to generate adversarial examples. In Proc. AAAI, 2018.
[6]
E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proc. FAccT, 2021.
[7]
B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. ?rndic, P. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In Proc. ECML/PKDD, 2013.
[8]
B. Biggio and F. Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84:317--331, 2018.
[9]
N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In Proc. IEEE S&P, 2017.
[10]
Y. Chen, Z. Ding, and D. Wagner. Continuous learning for android malware detection. In Proc. USENIX Security, 2023.
[11]
Chronicle. Virustotal. https://www.virustotal.com/, 2004--. Accessed 6/17/2019.
[12]
A. Datta, M. Fredrikson, K. Leino, K. Lu, S. Sen, and Z. Wang. Machine learning explainability and robustness: Connected at the hip. In Proc. KDD, 2021.
[13]
L. Demetrio, B. Biggio, G. Lagorio, F. Roli, and A. Armando. Explaining vulnerabilities of deep learning to adversarial malware binaries. In Proc. ITASEC, 2019.
[14]
B. G. Doan, S. Yang, P. Montague, O. De Vel, T. Abraham, S. Camtepe, S. S. Kanhere, E. Abbasnejad, and D. C. Ranasinghe. Feature-space bayesian adversarial learning improved malware detector robustness. In Proc. AAAI, 2023.
[15]
A. Feizollah, N. B. Anuar, R. Salleh, and A. W. A. Wahab. A review on feature selection in mobile malware detection. Digit. Investig., 13(C):22--37, Jun 2015.
[16]
M. Fernández-Delgado, M. Sirsat, E. Cernadas, S. Alawadi, S. Barro, and M. Febrero-Bande. An extensive experimental survey of regression methods. Neural Networks, 111:11--34, 2019.
[17]
M. Galovic, B. Bosanský, and V. Lisý. Improving robustness of malware classifiers using adversarial strings generated from perturbed latent representations. In Proc. NeurIPSW, 2021.
[18]
I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In Proc. ICLR, 2015.
[19]
J. Heo, S. Joo, and T. Moon. Fooling neural network interpretations via adversarial model manipulation. In Proc. NeurIPS, 2019.
[20]
Z. Huang, N. G. Marchant, K. Lucas, L. Bauer, O. Ohrimenko, and B. I. P. Rubinstein. Rs-del: Edit distance robustness certificates for sequence classifiers via randomized deletion. In Proc. NeurIPS, 2023.
[21]
I. Incer, M. Theodorides, S. Afroz, and D. Wagner. Adversarially robust malware detection using monotonic classification. In Proc. IWSPA, 2018.
[22]
B. Kolosnjaji, A. Demontis, B. Biggio, D. Maiorca, G. Giacinto, C. Eckert, and F. Roli. Adversarial malware binaries: Evading deep learning for malware detection in executables. In Proc. EUSIPCO, 2018.
[23]
J. Z. Kolter and M. A. Maloof. Learning to detect and classify malicious executables in the wild. Journal of Machine Learning Research, 2006.
[24]
H. Koo and M. Polychronakis. Juggling the gadgets: Binary-level code randomization using instruction displacement. In Proc. AsiaCCS, 2016.
[25]
M. Krcál, O. vec, M. Bálek, and O. Jaek. Deep convolutional malware classifiers can learn from raw executables and labels only. In Proc. ICLRW, 2018.
[26]
F. Kreuk, A. Barak, S. Aviv-Reuven, M. Baruch, B. Pinkas, and J. Keshet. Adversarial examples on discrete sequences for beating whole-binary malware detection. In Proc. NeurIPSW, 2018.
[27]
K. Lucas, S. Pai, W. Lin, L. Bauer, M. K. Reiter, and M. Sharif. Adversarial training for raw-binary malware classifiers. In Proc. USENIX Security, 2023.
[28]
K. Lucas, M. Sharif, L. Bauer, M. K. Reiter, and S. Shintre. Malware makeover: Breaking ML-based static analysis by modifying executable bytes. In Proc. AsiaCCS, 2021.
[29]
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In Proc. ICLR, 2018.
[30]
N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami. The limitations of deep learning in adversarial settings. In Proc. IEEE Euro S&P, 2016.
[31]
V. Pappas, M. Polychronakis, and A. D. Keromytis. Smashing the gadgets: Hindering return-oriented programming using in-place code randomization. In Proc. IEEE S&P, 2012.
[32]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825--2830, 2011.
[33]
F. Pierazzi, F. Pendlebury, J. Cortellazzi, and L. Cavallaro. Intriguing properties of adversarial ml attacks in the problem space. In Proc. IEEE S&P, 2020.
[34]
E. Raff, J. Barker, J. Sylvester, R. Brandon, B. Catanzaro, and C. K. Nicholas. Malware detection by eating a whole exe. In Proc. AAAIW, 2018.
[35]
M. Schultz, E. Eskin, F. Zadok, and S. Stolfo. Data mining methods for detection of new malicious executables. In Proc. IEEE S&P, 2001.
[36]
A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein. Adversarial training for free! In Proc. NeurIPS, 2019.
[37]
M. Sharif, K. Lucas, L. Bauer, M. K. Reiter, and S. Shintre. Optimization-guided binary diversification to mislead neural networks for malware detection. arXiv preprint, 2019.
[38]
K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proc. ICLRW, 2014.
[39]
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929--1958, jan 2014.
[40]
O. Suciu, S. E. Coull, and J. Johns. Exploring adversarial examples in malware detection. In Proc. AAAIW, 2018.
[41]
R. Sun, M. Xue, G. Tyson, T. Dong, S. Li, S. Wang, H. Zhu, S. Camtepe, and S. Nepal. Mate! are you really aware? an explainability-guided testing framework for robustness of malware detectors. In Proc. ESEC/FSE, 2023.
[42]
M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. Proc. ICML, 2017.
[43]
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In Proc. ICLR, 2014.
[44]
L. Tong, B. Li, C. Hajaj, C. Xiao, N. Zhang, and Y. Vorobeychik. Improving robustness of ml classifiers against realizable evasion attacks using conserved features. In Proc. USENIX Security, 2019.
[45]
Z. Wang. On the Feature Alignment of Deep Vision Models: Explainability and Robustness Connected At Hip. PhD thesis, Carnegie Mellon University, 2023.
[46]
Z. Wang, M. Fredrikson, and A. Datta. Robust models are more interpretable because attributions look normal. In Proc. ICML, 2021.
[47]
Z. Wang, P. Mardziel, A. Datta, and M. Fredrikson. Interpreting interpretations: Organizing attribution methods by criteria. In Proc. CVPRW, 2020.
[48]
E. Wong, L. Rice, and J. Z. Kolter. Fast is better than free: Revisiting adversarial training. In Proc. ICLR, 2020.
[49]
P. Yang, J. Chen, C.-J. Hsieh, J.-L. Wang, and M. I. Jordan. Greedy attack and gumbel attack: Generating adversarial examples for discrete data. Journal of Machine Learning Research, 21(43):1--36, 2020.
[50]
X. Zhang, N. Wang, H. Shen, S. Ji, X. Luo, and T. Wang. Interpretable deep learning under fire. In Proc. USENIX Security, 2020.
Index Terms
- Training Robust ML-based Raw-Binary Malware Detectors in Hours, not Months
Recommendations
REMEDII: Robust Malware Detection with Iterative and Intelligent Adversarial Training
Information Systems SecurityAbstractMalware detection traditionally relies on signature-based approaches, which suffer from limited generalization. To mitigate this issue, machine learning (ML)-based detection methods have been integrated with signature-based methods in recent ...
A novel method for improving the robustness of deep learning-based malware detectors against adversarial attacks
AbstractMalware is constantly evolving with rising concern for cyberspace. Deep learning-based malware detectors are being used as a potential solution. However, these detectors are vulnerable to adversarial attacks. The adversarial attacks manipulate ...
Graphical abstractDisplay Omitted
Highlights- An approach to combining adversarial attacks is proposed to analyse the robustness of malware detectors against attacks.
- Ten adversarial attacks are created to generate binary-encoded malicious samples, including the proposed combined ...
Towards Adversarially Superior Malware Detection Models: An Adversary Aware Proactive Approach using Adversarial Attacks and Defenses
AbstractThe android ecosystem (smartphones, tablets, etc.) has grown manifold in the last decade. However, the exponential surge of android malware is threatening the ecosystem. Literature suggests that android malware can be detected using machine and ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2024
5188 pages
ISBN:9798400706363
DOI:10.1145/3658644
- General Chairs:
- Bo Luo,
- Xiaojing Liao,
- Jun Xu,
- Program Chairs:
- Engin Kirda,
- David Lie
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 December 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
- Maof Prize for Outstanding Young Scientists
- United States-Israel Binational Science Foundation
- Defence Science and Technology Agency - Singapore
- National Science Foundation
- Intel Corporation
- Ministry of Innovation, Science & Technology, Israel
- Army Research Office
Conference
CCS '24
Sponsor:
CCS '24: ACM SIGSAC Conference on Computer and Communications Security
October 14 - 18, 2024
UT, Salt Lake City, USA
Acceptance Rates
Overall Acceptance Rate 1,261 of 6,999 submissions, 18%
Upcoming Conference
CCS '25
- Sponsor:
- sigsac
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 264Total Downloads
- Downloads (Last 12 months)264
- Downloads (Last 6 weeks)151
Reflects downloads up to 16 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in