research-article

Robustness of on-device models: adversarial attack to deep learning models on Android apps

Authors:

Chunyang ChenAuthors Info & Claims

ICSE-SEIP '21: Proceedings of the 43rd International Conference on Software Engineering: Software Engineering in Practice

Pages 101 - 110

https://doi.org/10.1109/ICSE-SEIP52600.2021.00019

Published: 17 December 2021 Publication History

Abstract

Deep learning has shown its power in many applications, including object detection in images, natural-language understanding, and speech recognition. To make it more accessible to end users, many deep learning models are now embedded in mobile apps. Compared to offloading deep learning from smartphones to the cloud, performing machine learning on-device can help improve latency, connectivity, and power consumption. However, most deep learning models within Android apps can easily be obtained via mature reverse engineering, while the models' exposure may invite adversarial attacks. In this study, we propose a simple but effective approach to hacking deep learning models using adversarial attacks by identifying highly similar pre-trained models from TensorFlow Hub. All 10 real-world Android apps in the experiment are successfully attacked by our approach. Apart from the feasibility of the model attack, we also carry out an empirical study that investigates the characteristics of deep learning models used by hundreds of Android apps on Google Play. The results show that many of them are similar to each other and widely use fine-tuning techniques to pre-trained models on the Internet.

References

[1]

https://github.com/XiaoMi/mace, 2020.

[2]

Apktool. https://ibotpeaches.github.io/Apktool/, 2020.

[3]

Caffe2. https://github.com/facebookarchive/caffe2, 2020.

[4]

Core ml. https://developer.apple.com/documentation/coreml, 2020.

[5]

Google mobile vision. https://developers.google.com/vision/, 2020.

[6]

Netron. https://lutzroeder.github.io/netron/, 2020.

[7]

Pytorch. https://pytorch.org, 2020.

[8]

Tencent ncnn. https://github.com/Tencent/ncnn, 2020.

[9]

Tensorflow. https://www.tensorflow.org, 2020.

[10]

Tensorflow hub. https://www.tensorflow.org/hub, 2020.

[11]

Tensorflow lite. https://https://www.tensorflow.org/lite, 2020.

[12]

M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308--318, 2016.

Digital Library

[13]

M. Bastian, S. Heymann, M. Jacomy, et al. Gephi: an open source software for exploring and manipulating networks. Icwsm, 8(2009):361--362, 2009.

[14]

J. Benesty, J. Chen, Y. Huang, and I. Cohen. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1--4. Springer, 2009.

Digital Library

[15]

V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.

[16]

T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer. Adversarial patch, 2017.

[17]

N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39--57. IEEE, 2017.

[18]

S. Chen, L. Fan, C. Chen, M. Xue, Y. Liu, and L. Xu. Guisquatting attack: Automated generation of android phishing apps. IEEE Transactions on Dependable and Secure Computing, 2019.

[19]

N. Çürükoğnlu and B. M. Özyildirim. Deep learning on mobile systems. In 2018 Innovations in Intelligent Systems and Applications Conference (ASYU), pages 1--4. IEEE, 2018.

[20]

X. Dai, I. Spasić, B. Meyer, S. Chapman, and F. Andres. Machine learning on mobile: An on-device inference app for skin cancer detection. In 2019 Fourth International Conference on Fog and Mobile Edge Computing (FMEC), pages 301--305. IEEE, 2019.

[21]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248--255. Ieee, 2009.

[22]

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

[23]

Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185--9193, 2018.

[24]

M. Fredrikson, S. Jha, and T. Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 1322--1333, 2015.

Digital Library

[25]

I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples, 2014.

[26]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016.

[27]

Y. He, G. Meng, K. Chen, X. Hu, and J. He. Towards privacy and security of deep learning systems: a survey. arXiv preprint arXiv:1911.12562, 2019.

[28]

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.

[29]

F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016.

[30]

U. Jang, X. Wu, and S. Jha. Objective metrics and gradient descent algorithms for adversarial examples in machine learning. In Proceedings of the 33rd Annual Computer Security Applications Conference, ACSAC 2017, page 262--277, 2017.

Digital Library

[31]

A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.

[32]

Y. Li, J. Hua, H. Wang, C. Chen, and Y. Liu. Deepbackdoor: Black-box backdoor attack on deep learning models through neural payload injection. In The 43rd International Conference on Software Engineering, 2021.

[33]

L. Ma, F. Juefei-Xu, F. Zhang, J. Sun, M. Xue, B. Li, C. Chen, T. Su, L. Li, Y. Liu, et al. Deepgauge: Multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pages 120--131, 2018.

Digital Library

[34]

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks, 2017.

[35]

S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. Deepfool: A simple and accurate method to fool deep neural networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016.

[36]

A. B. Nassif, I. Shahin, I. Attili, M. Azzeh, and K. Shaalan. Speech recognition using deep neural networks: A systematic review. IEEE Access, 7:19143--19165, 2019.

[37]

K. Ota, M. S. Dao, V. Mezaris, and F. G. D. Natale. Deep learning for mobile multimedia: A survey. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 13(3s):1--22, 2017.

[38]

S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345--1359, 2009.

Digital Library

[39]

A. R. Pathak, M. Pandey, and S. Rautaray. Application of deep learning for object detection. Procedia computer science, 132:1706--1717, 2018.

[40]

J. Rony, L. G. Hafemann, L. S. Oliveira, I. Ben Ayed, R. Sabourin, and E. Granger. Decoupling direction and norm for efficient gradient-based l2 adversarial attacks and defenses. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019.

[41]

P. Sanabria, J. I. Benedetto, A. Neyem, J. Navon, and C. Poellabauer. Code offloading solutions for audio processing in mobile healthcare applications: a case study. In 2018 IEEE/ACM 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft), pages 117--121. IEEE, 2018.

Digital Library

[42]

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510--4520, 2018.

[43]

Z. Sun, R. Sun, and L. Lu. Mind your weight (s): A large-scale study on insufficient machine learning model protection in mobile apps. arXiv preprint arXiv:2002.07687, 2020.

[44]

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818--2826, 2016.

[45]

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.

[46]

F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart. Stealing machine learning models via prediction apis. In 25th {USENIX} Security Symposium ({USENIX} Security 16), pages 601--618, 2016.

[47]

B. Wang, Y. Yao, B. Viswanath, H. Zheng, and B. Y. Zhao. With great training comes great vulnerability: Practical attacks against transfer learning. In 27th {USENIX} Security Symposium ({USENIX} Security 18), pages 1281--1297, 2018.

[48]

J. Wang, B. Cao, P. Yu, L. Sun, W. Bao, and X. Zhu. Deep learning towards mobile applications. In 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), pages 1385--1393. IEEE, 2018.

[49]

M. Xu, J. Liu, Y. Liu, F. X. Lin, Y. Liu, and X. Liu. A first look at deep learning apps on smartphones. In The World Wide Web Conference, pages 2125--2136, 2019.

Digital Library

[50]

Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5753--5763. Curran Associates, Inc., 2019.

[51]

T. Young, D. Hazarika, S. Poria, and E. Cambria. Recent trends in deep learning based natural language processing. ieee Computational intelligenCe magazine, 13(3):55--75, 2018.

[52]

H. Zhang and J. Wang. Defense against adversarial attacks using feature scattering-based adversarial training. In Advances in Neural Information Processing Systems, pages 1831--1841, 2019.

[53]

J. Zhang and C. Li. Adversarial examples: Opportunities and challenges. IEEE transactions on neural networks and learning systems, 2019.

[54]

Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint arXiv:1506.06724, 2015.

Cited By

Chen HBabar M(2024)Security for Machine Learning-based Software Systems: A Survey of Threats, Practices, and ChallengesACM Computing Surveys10.1145/363853156:6(1-38)Online publication date: 23-Feb-2024
https://dl.acm.org/doi/10.1145/3638531
Zhou MGao XWu JLiu KSun HLi LRoychoudhury APaiva AAbreu RStorey M(2024)Investigating White-Box Attacks for On-Device ModelsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639144(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639144
Ren PZuo CLiu XDiao WZhao QGuo SRoychoudhury APaiva AAbreu RStorey M(2024)DEMISTIFY: Identifying On-device Machine Learning Models Stealing and Reuse Vulnerabilities in Mobile AppsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623325(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623325
Show More Cited By

Index Terms

Robustness of on-device models: adversarial attack to deep learning models on Android apps
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
    2. Machine learning approaches
      1. Neural networks
2. Security and privacy

Index terms have been assigned to the content through auto-classification.

Recommendations

A First Look at Deep Learning Apps on Smartphones
WWW '19: The World Wide Web Conference

To bridge the knowledge gap between research and practice, we present the first empirical study on 16,500 the most popular Android apps, demystifying how smartphone apps exploit deep learning in the wild. To this end, we build a new static tool that ...
An Explorative Study of the Mobile App Ecosystem from App Developers' Perspective
WWW '17: Proceedings of the 26th International Conference on World Wide Web

With the prevalence of smartphones, app markets such as Apple App Store and Google Play has become the center stage in the mobile app ecosystem, with millions of apps developed by tens of thousands of app developers in each major market. This paper ...
A First Look at On-device Models in iOS Apps
Powered by the rising popularity of deep learning techniques on smartphones, on-device deep learning models are being used in vital fields such as finance, social media, and driving assistance. Because of the transparency of the Android platform and the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE-SEIP '21: Proceedings of the 43rd International Conference on Software Engineering: Software Engineering in Practice

May 2021

405 pages

ISBN:9780738146690

Conference Chairs:
Sigrid Eldh
Ericsson, Sweden
,
Davide Falessi
California Polytechnic State University

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

IEEE Press

Publication History

Published: 17 December 2021

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '21

Sponsor:

SIGSOFT

ICSE '21: 43rd International Conference on Software Engineering

May 25 - 28, 2021

Virtual Event, Spain

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
41
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 28 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen HBabar M(2024)Security for Machine Learning-based Software Systems: A Survey of Threats, Practices, and ChallengesACM Computing Surveys10.1145/363853156:6(1-38)Online publication date: 23-Feb-2024
https://dl.acm.org/doi/10.1145/3638531
Zhou MGao XWu JLiu KSun HLi LRoychoudhury APaiva AAbreu RStorey M(2024)Investigating White-Box Attacks for On-Device ModelsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639144(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639144
Ren PZuo CLiu XDiao WZhao QGuo SRoychoudhury APaiva AAbreu RStorey M(2024)DEMISTIFY: Identifying On-device Machine Learning Models Stealing and Reuse Vulnerabilities in Mobile AppsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623325(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623325
Hu HHuang YChen QZhuo TChen C(2023)A First Look at On-device Models in iOS AppsACM Transactions on Software Engineering and Methodology10.1145/361717733:1(1-30)Online publication date: 23-Nov-2023
https://dl.acm.org/doi/10.1145/3617177
Zhou MGao XWu JGrundy JChen XChen CLi LJust RFraser G(2023)ModelObfuscator: Obfuscating Model Information to Protect Deployed ML-Based SystemsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598113(1005-1017)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598113
Sang YHuang YHuang SCui H(2023)Beyond the Model: Data Pre-processing Attack to Deep Learning Models in Android AppsProceedings of the 2023 Secure and Trustworthy Deep Learning Systems Workshop10.1145/3591197.3591308(1-9)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3591197.3591308
Wang KHe QChen FChen CHuang FJin HYang Y(2023)FlexiFed: Personalized Federated Learning for Edge Clients with Heterogeneous Model ArchitecturesProceedings of the ACM Web Conference 202310.1145/3543507.3583347(2979-2990)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583347
Le TChen HBabar M(2022)A Survey on Data-driven Software Vulnerability Assessment and PrioritizationACM Computing Surveys10.1145/352975755:5(1-39)Online publication date: 3-Dec-2022
https://dl.acm.org/doi/10.1145/3529757
Zheng HChen ZDu TZhang XCheng YJi SWang JYu YChen JDwyer MDamian DZeller A(2022)NeuronFairProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510123(1519-1531)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510123
Su YLiu ZChen CWang JWang QSpinellis DGousios GChechik MDi Penta M(2021)OwlEyes-online: a fully automated platform for detecting and localizing UI display issuesProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3473109(1500-1504)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3468264.3473109

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents