research-article

Code obfuscation against symbolic execution attacks

Authors:

Sebastian Banescu,

Christian Collberg,

Alexander PretschnerAuthors Info & Claims

ACSAC '16: Proceedings of the 32nd Annual Conference on Computer Security Applications

Pages 189 - 200

https://doi.org/10.1145/2991079.2991114

Published: 05 December 2016 Publication History

Abstract

Code obfuscation is widely used by software developers to protect intellectual property, and malware writers to hamper program analysis. However, there seems to be little work on systematic evaluations of effectiveness of obfuscation techniques against automated program analysis. The result is that we have no methodical way of knowing what kinds of automated analyses an obfuscation method can withstand.

This paper addresses the problem of characterizing the resilience of code obfuscation transformations against automated symbolic execution attacks, complementing existing works that measure the potency of obfuscation transformations against human-assisted attacks through user studies. We evaluated our approach over 5000 different C programs, which have each been obfuscated using existing implementations of obfuscation transformations. The results show that many existing obfuscation transformations, such as virtualization, stand little chance of withstanding symbolic-execution based deobfuscation. A crucial and perhaps surprising observation we make is that symbolic-execution based deobfuscators can easily deobfuscate transformations that preserve program semantics. On the other hand, we present new obfuscation transformations that change program behavior in subtle yet acceptable ways, and show that they can render symbolic-execution based deobfuscation analysis ineffective in practice.

References

[1]

S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, P. McMinn, et al. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software, 86(8):1978--2001, 2013.

Digital Library

[2]

S. Anand, P. Godefroid, and N. Tillmann. Demand-driven compositional symbolic execution. In Tools and Algorithms for the Construction and Analysis of Systems, pages 367--381. Springer, 2008.

Digital Library

[3]

B. Anckaert, M. Madou, B. De Sutter, B. De Bus, K. De Bosschere, and B. Preneel. Program obfuscation: a quantitative approach. In Proceedings of the 2007 ACM workshop on Quality of protection, pages 15--20. ACM, 2007.

Digital Library

[4]

D. Aucsmith. Tamper resistant software: An implementation. In Information Hiding, pages 317--333. Springer, 1996.

Digital Library

[5]

T. Avgerinos, S. K. Cha, A. Rebert, E. J. Schwartz, M. Woo, and D. Brumley. Automatic exploit generation. Communications of the ACM, 57(2):74--84, 2014.

Digital Library

[6]

S. Banescu, M. Ochoa, and A. Pretschner. A framework for measuring software obfuscation resilience against automated attacks. In Software Protection (SPRO), 2015 IEEE/ACM 1st International Workshop on, pages 45--51. IEEE, 2015.

Digital Library

[7]

S. Banescu, A. Pretschner, D. Battré, S. Cazzulani, R. Shield, and G. Thompson. Software-based protection against changeware. In Proceedings of the 5th ACM Conference on Data and Application Security and Privacy, pages 231--242. ACM, 2015.

Digital Library

[8]

B. Barak, O. Goldreich, R. Impagliazzo, S. Rudich, A. Sahai, S. Vadhan, and K. Yang. On the (im) possibility of obfuscating programs. In Advances in Cryptology CRYPTO 2001, pages 1--18. Springer, 2001.

Digital Library

[9]

C. W. Barrett, R. Sebastiani, S. A. Seshia, and C. Tinelli. Satisfiability modulo theories. Handbook of satisfiability, 185:825--885, 2009.

[10]

C. Basile, S. Di Carlo, T. Herlea, V. Business, J. Nagra, and B. Wyseur. Towards a formal model for software tamper resistance. In Second International Workshop on Remote Entrusting (ReTtust 2009), volume 16.

[11]

C. Cadar, D. Dunbar, and D. R. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, 2008.

Digital Library

[12]

C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE: Automatically generating inputs of death. In Proceedings of the 13th ACM Conference on Computer and Communications Security, CCS '06, pages 322--335, New York, NY, USA, 2006. ACM. 00041.

Digital Library

[13]

L. Cavallaro, P. Saxena, and R. Sekar. On the limits of information flow techniques for malware analysis and containment. In Detection of Intrusions and Malware, and Vulnerability Assessment, pages 143--163. Springer, 2008.

Digital Library

[14]

M. Ceccato, M. Di Penta, J. Nagra, P. Falcarin, F. Ricca, M. Torchiano, and P. Tonella. The effectiveness of source code obfuscation: an experimental assessment. In Program Comprehension, 2009. ICPC'09. IEEE 17th International Conference on, pages 178--187. IEEE, 2009.

[15]

M. Ceccato, M. D. Penta, P. Falcarin, F. Ricca, M. Torchiano, and P. Tonella. A family of experiments to assess the effectiveness and efficiency of source code obfuscation techniques. Empirical Software Engineering, 19(4):1040--1074, Feb. 2013.

Digital Library

[16]

H. Chang and M. J. Atallah. Protecting software code by guards. In Security and privacy in digital rights management, pages 160--175. Springer, 2001.

Digital Library

[17]

V. Chipounov, V. Kuznetsov, and G. Candea. S2E: A Platform for In-vivo Multi-path Analysis of Software Systems. ASPLOS XVI, pages 265--278, New York, NY, USA, 2011. ACM.

Digital Library

[18]

C. Collberg, S. Martin, J. Myers, and J. Nagra. Distributed application tamper detection via continuous software updates. In Proceedings of the 28th Annual Computer Security Applications Conference, ACSAC '12, pages 319--328, New York, NY, USA, 2012. ACM.

Digital Library

[19]

C. Collberg and J. Nagra. Surreptitious software. Upper Saddle River, NJ: Addision-Wesley Professional, 2010.

[20]

C. Collberg, C. Thomborson, and D. Low. A taxonomy of obfuscating transformations. Technical report, Department of Computer Science, The University of Auckland, New Zealand, 1997.

[21]

C. Collberg, C. Thomborson, and D. Low. Manufacturing cheap, resilient, and stealthy opaque constructs. In Proceedings of the 25th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, POPL '98, pages 184--196, New York, NY, USA, 1998. ACM.

Digital Library

[22]

K. Coogan, G. Lu, and S. Debray. Deobfuscation of virtualization-obfuscated software: A semantics-based approach. In Proceedings of the 18th ACM Conference on Computer and Communications Security, CCS '11, pages 275--284, New York, NY, USA, 2011. ACM.

Digital Library

[23]

M. Dalla Preda. Code obfuscation and malware detection by abstract interpretation. PhD thesis, University of Verona, 2007.

[24]

M. Dalla Preda and R. Giacobazzi. Control code obfuscation by abstract interpretation. In Third IEEE International Conference on Software Engineering and Formal Methods., pages 301--310. IEEE, 2005.

Digital Library

[25]

C. Eagle. The IDA pro book: the unofficial guide to the world's most popular disassembler. No Starch Press, 2011.

Digital Library

[26]

S. Forrest, A. Somayaji, and D. H. Ackley. Building diverse computer systems. In Operating Systems, 1997., The Sixth Workshop on Hot Topics in, pages 67--72. IEEE, 1997.

Digital Library

[27]

M. Franz. E unibus pluram: massive-scale software diversity as a defense mechanism. In Proceedings of the 2010 workshop on New security paradigms, pages 7--16. ACM, 2010.

Digital Library

[28]

V. Ganesh and D. L. Dill. A decision procedure for bit-vectors and arrays. In Computer Aided Verification, pages 519--531. Springer, 2007.

Digital Library

[29]

S. Garg, C. Gentry, S. Halevi, M. Raykova, A. Sahai, and B. Waters. Candidate indistinguishability obfuscation and functional encryption for all circuits. In Proc. of the 54th Annual Symp. on Foundations of Computer Science, pages 40--49, 2013.

Digital Library

[30]

L. E. Garner. On the Collatz 3n + 1 algorithm. Proceedings of the American Mathematical Society, 82(1):19--22, 1981.

[31]

I. P. Gent, E. MacIntyre, P. Prosser, T. Walsh, et al. The constrainedness of search. In AAAI/IAAI, Vol. 1, pages 246--252, 1996.

Digital Library

[32]

P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '05, pages 213--223, New York, NY, USA, 2005.

Digital Library

[33]

P. Godefroid, M. Y. Levin, and D. Molnar. Sage: whitebox fuzzing for security testing. Queue, 10(1):20, 2012.

Digital Library

[34]

P. Godefroid, M. Y. Levin, D. A. Molnar, et al. Automated whitebox fuzz testing. In NDSS, volume 8, pages 151--166, 2008.

[35]

Y. Guillot and A. Gazet. Automatic binary deobfuscation. Journal in computer virology, 6(3):261--276, 2010.

[36]

B. Horne, L. Matheson, C. Sheehan, and R. E. Tarjan. Dynamic self-checking techniques for improved tamper resistance. In Security and privacy in digital rights management, pages 141--159. Springer, 2002.

Digital Library

[37]

P. Junod, J. Rinaldini, J. Wehrli, and J. Michielin. Obfuscator-LLVM - software protection for the masses. In B. Wyseur, editor, Proceedings of the IEEE/ACM 1st International Workshop on Software Protection, SPRO'15, Firenze, Italy, May 19th, 2015, pages 3--9. IEEE, 2015.

Digital Library

[38]

J. Kinder. Towards static analysis of virtualization-obfuscated binaries. In 19th Working Conference on Reverse Engineering (WCRE), pages 61--70, Oct 2012.

Digital Library

[39]

J. C. King. Symbolic execution and program testing. Communications of the ACM, 19(7):385--394, 1976.

Digital Library

[40]

McAfee. McAfee Labs Threats Report. Technical Report March, 2016. http://www.mcafee.com/us/resources/reports/rp-quarterly-threats-mar-2016.pdf.

[41]

I. Mironov and L. Zhang. Applications of sat solvers to cryptanalysis of hash functions. In Theory and Applications of Satisfiability Testing-SAT 2006, pages 102--115. Springer, 2006.

Digital Library

[42]

G. Naumovich and N. Memon. Preventing piracy, reverse engineering, and tampering. Computer, (7):64--71, 2003.

Digital Library

[43]

J. Qiu, B. Yadegari, B. Johannesmeyer, S. Debray, and X. Su. Identifying and understanding self-checksumming defenses in software. In Proceedings of the 5th ACM Conference on Data and Application Security and Privacy, pages 207--218. ACM, 2015.

Digital Library

[44]

R. Rolles. Program Synthesis in Reverse Engineering. http://www.nosuchcon.org/talks/2014/D1_01_Rolf_Rolles_Program_Synthesis_in_reverse_Engineering.pdf, 2014. NoSuchCon 2014, Accessed:2016-05-24.

[45]

F. Saudel and J. Salwan. Triton: A dynamic symbolic execution framework. In Symposium sur la sécurité des technologies de l'information et des communications, SSTIC, France, Rennes, June 3--5 2015, pages 31--54. SSTIC, 2015.

[46]

S. Schrittwieser, S. Katzenbeisser, J. Kinder, G. Merzdovnik, and E. Weippl. Protecting software through obfuscation: Can it keep pace with progress in code analysis? ACM Computing Surveys (CSUR), 49(1):4, 2016.

Digital Library

[47]

E. J. Schwartz, T. Avgerinos, and D. Brumley. All you ever wanted to know about dynamic taint analysis and forward symbolic execution (but might have been afraid to ask). In Security and Privacy (SP), 2010 IEEE Symposium on, pages 317--331. IEEE, 2010.

Digital Library

[48]

M. Sharif, A. Lanzi, J. Giffin, and W. Lee. Automatic reverse engineering of malware emulators. In Security and Privacy, 2009 30th IEEE Symposium on, pages 94--109, May 2009.

Digital Library

[49]

M. I. Sharif, A. Lanzi, J. T. Giffin, and W. Lee. Impeding malware analysis using conditional code obfuscation. In NDSS, 2008.

[50]

Y. Shoshitaishvili, R. Wang, C. Hauser, C. Kruegel, and G. Vigna. Firmalice - automatic detection of authentication bypass vulnerabilities in binary firmware. 2015.

[51]

D. Song, D. Brumley, H. Yin, J. Caballero, I. Jager, M. G. Kang, Z. Liang, J. Newsome, P. Poosankam, and P. Saxena. Bitblaze: A new approach to computer security via binary analysis. In Information systems security, pages 1--25. Springer, 2008.

Digital Library

[52]

Symantec Corporation. Internet Security Threat Report 2016. Technical Report April, 2016. https://www.symantec.com/content/dam/symantec/docs/reports/istr-21-2016-en.pdf.

[53]

S. Udupa, S. Debray, and M. Madou. Deobfuscation: reverse engineering obfuscated code. In 12th Working Conference on Reverse Engineering, 2005.

Digital Library

[54]

M. Varia. Studies in program obfuscation. PhD thesis, School of Computer Science, Tel Aviv University, 2010.

Digital Library

[55]

T. Wang, T. Wei, G. Gu, and W. Zou. Taintscope: A checksum-aware directed fuzzing tool for automatic software vulnerability detection. In Security and privacy (SP), 2010 IEEE symposium on, pages 497--512. IEEE, 2010.

Digital Library

[56]

Z. Wang, J. Ming, C. Jia, and D. Gao. Linear obfuscation to combat symbolic execution. In Computer Security-ESORICS 2011, pages 210--226. Springer, 2011.

Digital Library

[57]

B. Yadegari and S. Debray. Symbolic execution of obfuscated code. In Proceedings of the 22Nd ACM SIGSAC Conference on Computer and Communications Security, ser. CCS, volume 15, pages 732--744, 2015.

Digital Library

[58]

B. Yadegari, B. Johannesmeyer, B. Whitely, and S. Debray. A generic approach to automatic deobfuscation of executable code. In Security and Privacy (SP), 2015 IEEE Symposium on, pages 674--691. IEEE, 2015.

Digital Library

Cited By

Jin HLee JYang SKim KLee D(2024)A Framework to Quantify the Quality of Source Code ObfuscationApplied Sciences10.3390/app1412505614:12(5056)Online publication date: 10-Jun-2024
https://doi.org/10.3390/app14125056
De Sutter BSchrittwieser SCoppens BKochberger P(2024)Evaluation Methodologies in Software Protection ResearchACM Computing Surveys10.1145/3702314Online publication date: 2-Nov-2024
https://doi.org/10.1145/3702314
Mariano BWang ZPailoor SCollberg CDillig I(2024)Control-Flow Deobfuscation using Trace-Informed Compositional Program SynthesisProceedings of the ACM on Programming Languages10.1145/36897898:OOPSLA2(2211-2241)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689789
Show More Cited By

Recommendations

Obfuscation: The Hidden Malware

A cyberwar exists between malware writers and antimalware researchers. At this war's heart rages a weapons race that originated in the 80s with the first computer virus. Obfuscation is one of the latest strategies to camouflage the telltale signs of ...
Linear obfuscation to combat symbolic execution
ESORICS'11: Proceedings of the 16th European conference on Research in computer security

Trigger-based code (malicious in many cases, but not necessarily) only executes when specific inputs are received. Symbolic execution has been one of the most powerful techniques in discovering such malicious code and analyzing the trigger condition. We ...
The power of obfuscation techniques in malicious JavaScript code: A measurement study
MALWARE '12: Proceedings of the 2012 7th International Conference on Malicious and Unwanted Software (MALWARE)

JavaScript based attacks have been reported as the top Internet security threats in recent years. Since most of the Internet users rely on anti-virus software to protect themselves from malicious JavaScript code, attackers exploit JavaScript obfuscation ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACSAC '16: Proceedings of the 32nd Annual Conference on Computer Security Applications

December 2016

614 pages

ISBN:9781450347716

DOI:10.1145/2991079

Conference Chair:
Stephen Schwab
USC Information Sciences Institute
,
Program Chairs:
Wil Robertson
Northeastern University
,
Davide Balzarotti
Eurecom

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

ACSA: Applied Computing Security Assoc

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 December 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

ACSAC '16

Sponsor:

ACSA

ACSAC '16: 2016 Annual Computer Security Applications Conference

December 5 - 8, 2016

California, Los Angeles, USA

Acceptance Rates

Overall Acceptance Rate 104 of 497 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

105
Total Citations
View Citations
1,569
Total Downloads

Downloads (Last 12 months)128
Downloads (Last 6 weeks)13

Reflects downloads up to 16 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jin HLee JYang SKim KLee D(2024)A Framework to Quantify the Quality of Source Code ObfuscationApplied Sciences10.3390/app1412505614:12(5056)Online publication date: 10-Jun-2024
https://doi.org/10.3390/app14125056
De Sutter BSchrittwieser SCoppens BKochberger P(2024)Evaluation Methodologies in Software Protection ResearchACM Computing Surveys10.1145/3702314Online publication date: 2-Nov-2024
https://doi.org/10.1145/3702314
Mariano BWang ZPailoor SCollberg CDillig I(2024)Control-Flow Deobfuscation using Trace-Informed Compositional Program SynthesisProceedings of the ACM on Programming Languages10.1145/36897898:OOPSLA2(2211-2241)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689789
Stevens KErdemir MZhang HKim TPearce P(2024)BluePrint: Automatic Malware Signature Generation for Internet ScanningProceedings of the 27th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3678890.3678923(197-214)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3678890.3678923
See RRöbert KFischer M(2024)Encrypted Endpoints: Defending Online Services from Illegitimate Bot AutomationProceedings of the 27th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3678890.3678918(166-180)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3678890.3678918
Black CScott-Hayward S(2024)Defeating Data Plane Attacks With Program ObfuscationIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2023.327793921:3(1317-1330)Online publication date: May-2024
https://doi.org/10.1109/TDSC.2023.3277939
Li GYu MFang DLi GMeng XJiang JHuang W(2024)X-MBA: Towards Heterogeneous Mixed Boolean-Arithmetic DeobfuscationMILCOM 2024 - 2024 IEEE Military Communications Conference (MILCOM)10.1109/MILCOM61039.2024.10773656(1082-1087)Online publication date: 28-Oct-2024
https://doi.org/10.1109/MILCOM61039.2024.10773656
Kitaoka TKanzaki YIshio TShimari KMatsumoto K(2024)Initial Investigation of Behavioral Changes of Obfuscated Programs Caused by Code Optimization2024 IEEE 35th International Symposium on Software Reliability Engineering Workshops (ISSREW)10.1109/ISSREW63542.2024.00059(109-110)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ISSREW63542.2024.00059
Wang HLiu ZWang SWang YTang QNie SWu S(2024)Are We There Yet? Filling the Gap Between Binary Similarity Analysis and Binary Software Composition Analysis2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P)10.1109/EuroSP60621.2024.00034(506-523)Online publication date: 8-Jul-2024
https://doi.org/10.1109/EuroSP60621.2024.00034
Yang ZChen HQiu Z(2024)A Design of Code Protection Scheme Based on the Combination of OPTEE and Encrypted File SystemNetwork Simulation and Evaluation10.1007/978-981-97-4522-7_10(148-162)Online publication date: 2-Aug-2024
https://doi.org/10.1007/978-981-97-4522-7_10
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents