review-article

Free access

Fuzzing: hack, art, and science

Author:

Patrice GodefroidAuthors Info & Claims

Communications of the ACM, Volume 63, Issue 2

Pages 70 - 76

https://doi.org/10.1145/3363824

Published: 22 January 2020 Publication History

All formats PDF

Abstract

Reviewing software testing techniques for finding security vulnerabilities.

References

[1]

Bastani, O., Sharma, R., Aiken, A. and Liang, P. Synthesizing program input grammars. In Proceedings of the 38^th ACM SIGPLAN Conf. Programming Language Design and Implementation, 2017, 95--110.

Google Scholar

[2]

Bounimova, E., Godefroid, P., and Molnar, D. Billions and billions of constraints: Whitebox fuzz testing in production. In Proceedings of 35^th Intern. Conf. Software Engineering, (San Francisco, May 2013), 122--131.

Crossref

Google Scholar

[3]

Cadar, C., Dunbar, D., and Engler, D. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of OSDI'08 (Dec 2008).

Google Scholar

[4]

Cadar, C. and Engler, D. Execution generated test cases: How to make systems code crash itself. In Proceedings of 12^th Intern. SPIN Workshop on Model Checking of Software 3639 (San Francisco, CA, Aug. 2005) Lecture Notes in Computer Science, Springer-Verlag.

Digital Library

Google Scholar

[5]

Chipounov, V., Kuznetsov, V. and Candea, G. S2E: A platform for in-vivo multi-path analysis of software systems. In Proceedings of ASPLOS'2011.

Google Scholar

[6]

Claessen, K. and Hughes, J. QuickCheck: A lightweight tool for random testing of Haskell programs. In Proceedings of ICFP'2000.

Digital Library

Google Scholar

[7]

de Moura, L. and Bjorner, N. Z3: An Efficient SMT Solver. In Proceedings of 14^th Intern. Conf. Tools and Algorithms for the Construction and Analysis of Systems 4963 (Budapest, April 2008), 337--340. Lecture Notes in Computer Science, Springer-Verlag.

Crossref

Google Scholar

[8]

Forrester, J.E. and Miller, B.P. An empirical study of the robustness of Windows NT applications using random testing. In Proceedings of the 4^th USENIX Windows System Symp., Seattle, (Aug. 2000).

Google Scholar

[9]

Gallagher, T., Jeffries, B., and Landauer, L. Hunting Security Bugs, Microsoft Press, 2006.

Google Scholar

[10]

Ganesh, V., Leek, T., and Rinard. M. Taint-based directed whitebox fuzzing. In Proceedings of ICSE '2009.

Google Scholar

[11]

Godefroid, P. Higher-order test generation. In Proceedings of ACM SIGPLAN 2011 Conf. Programming Language, Design and Implementation (San Jose, June 2011). 258--269.

Digital Library

Google Scholar

[12]

Godefroid, P., Kiezun, A., and Levin, M.Y. Grammar-based whitebox fuzzing. In Proceedings of ACM SIGPLAN 2008 Conf. Programming Language Design and Implementation, (Tucson, AZ, USA, June 2008), 206--215.

Digital Library

Google Scholar

[13]

Godefroid, P., Klarlund, N., and Sen, K. DART: Directed automated random testing. In Proceedings of ACM SIGPLAN 2005 Conf. Programming Language Design and Implementation (Chicago, IL, June 2005). 213--223.

Digital Library

Google Scholar

[14]

Godefroid, P., Levin, M.Y., and Molnar, D. Automated whitebox fuzz testing. In Proceedings of Network and Distributed Systems Security (San Diego, Feb. 2008), 151--166.

Google Scholar

[15]

Godefroid, P., Levin, M.Y., and Molnar, D. SAGE: Whitebox fuzzing for security testing. Commun. ACM 55, 3 (Mar. 2012), 40--44.

Digital Library

Google Scholar

[16]

Godefroid, P., Peleg, H., and Singh, R. Learn&Fuzz: Machine Learning for Input Fuzzing. In Proceedings of the 32^nd IEEE/ACM Intern. Conf. Automated Software Engineering, Nov. 2017.

Digital Library

Google Scholar

[17]

Hanford, K.V. Automatic generation of test cases. IBM Systems J, 9, 4 (1970).

Digital Library

Google Scholar

[18]

Hoare, C.A.R. An axiomatic approach to computer programming. Commun. ACM 12, 10 (1969), 576--580.

Digital Library

Google Scholar

[19]

Holler, C., Herzig, K., and Zeller, A. Fuzzing with code fragments. In Proceedings of the 21st USENIX Security Symp., 2012.

Digital Library

Google Scholar

[20]

Höschele, M. and Zeller, A. Mining input grammars with AUTOGRAM. In Proceedings of ICSE-C'2017, 31--34.

Google Scholar

[21]

Howard, M. and Lipner, S. The Security Development Lifecycle. Microsoft Press, 2006.

Digital Library

Google Scholar

[22]

King, J.C. Symbolic execution and program Ttesting. J. ACM 19, 7 (1976), 385--394.

Google Scholar

[23]

Klees, G.T., Ruef, A., Cooper, B., Wei, S., and Hicks, M. Evaluating fuzz testing. In Proceedings of the ACM Conf. Computer and Communications Security, 2018.

Digital Library

Google Scholar

[24]

Lämmel, R. and Schulte, W. Controllable combinatorial coverage in grammar-based testing. In Proceedings of TestCom, 2006.

Digital Library

Google Scholar

[25]

Majumdar, R. and Xu, R. Directed test generation using symbolic grammars. In Proceedings of ASE, 2007.

Google Scholar

[26]

Maurer, P.M. Generating test data with enhanced context-free grammars. IEEE Software 7, 4 (1990).

Digital Library

Google Scholar

[27]

McMinn, P. Search-based software test data generation: A survey. Software Testing, Verification and Reliability 14, 2 (2004).

Crossref

Google Scholar

[28]

Pasareanu, C. S., Visser, W., Bushnell, D., Geldenhuys, J., Mehlitz, P., and Rungta, N. Symbolic pathFinder: Integrating symbolic execution with model checking for Java bytecode analysis. Automated Software Engineering, 2013, 20:391--425.

Crossref

Google Scholar

[29]

Peach Fuzzer; http://www.peachfuzzer.com/.

Google Scholar

[30]

Project Springfield; https://www.microsoft.com/springfield/, 2015.

Google Scholar

[31]

Protos; http://www.ee.oulu.fi/research/ouspg/protos/.

Google Scholar

[32]

SPIKE Fuzzer; http://resources.infosecinstitute.com/fuzzer-automation-with-spike/.

Google Scholar

[33]

Stephens, N. et al. Driller: Augmenting fuzzing through selective symbolic execution. In Proceedings of Network and Distributed Systems Security, 2016.

Google Scholar

[34]

Sulley; https://github.com/OpenRCE/sulley.

Google Scholar

[35]

Sutton, M., Greene, A., and Amini, P. Fuzzing: Brute Force Vulnerability Discovery. Addison-Wesley, 2007.

Digital Library

Google Scholar

[36]

Utting, M., Pretschner, A., and Legeard, B. A Taxonomy of model-based testing approaches. Intl. J. Software Testing, Verification and Reliability 22, 5 (2012).

Digital Library

Google Scholar

[37]

Walker, M. et al. DARPA Cyber Grand Challenge, 2016; http://archive.darpa.mil/cybergrandchallenge/.

Google Scholar

[38]

Yang, X., Chen, Y., Eide, E., and Regehr, J. Finding and understanding bugs in C compilers. In Proceedings of PLDI'2011.

Google Scholar

[39]

Yun, I., Lee, S., Xu, M., Jang, Y., and Kim, T. Qsym: A practical concolic execution engine tailored for hybrid fuzzing. In Proceedings of the 27^th USENIX Security Symp., 2018.

Google Scholar

[40]

Zalewski, M. AFL (American Fuzzy Lop), 2015; http://lcamtuf.coredump.cx/afl/

Google Scholar

Cited By

View all

Kummita SZhang ZBodden EWei SFilkov VRay BZhou M(2024)Visualizing and Understanding the Internals of FuzzingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695284(2199-2204)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695284
Kummita SMiao MBodden EWei SBöhme MNoller YSzekeres L(2024)Visualization Task Taxonomy to Understand the Fuzzing Internals (Registered Report)Proceedings of the 3rd ACM International Fuzzing Workshop10.1145/3678722.3685530(13-22)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3678722.3685530
Yi XLi GHuang WLin XLi JLiu Y(2024)LateBA: Latent Backdoor Attack on Deep Bug Search via Infrequent Execution CodesProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3674806(427-436)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3674806
Show More Cited By

Index Terms

Fuzzing: hack, art, and science
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Grammar-based whitebox fuzzing
PLDI '08: Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation

Whitebox fuzzing is a form of automatic dynamic test generation, based on symbolic execution and constraint solving, designed for security testing of large applications. Unfortunately, the current effectiveness of whitebox fuzzing is limited when ...
Fuzzing: Brute Force Vulnerability Discovery
Fuzzing for CPS Mutation Testing
ASE '23: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering

Mutation testing can help reduce the risks of releasing faulty software. For such reason, it is a desired practice for the development of embedded software running in safety-critical cyber-physical systems (CPS). Unfortunately, state-of-the-art test data ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

Communications of the ACM Volume 63, Issue 2

February 2020

80 pages

ISSN:0001-0782

EISSN:1557-7317

DOI:10.1145/3380852

Editor:
Andrew A. Chien
Association for Computing Machinery, New York, NY

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 January 2020

Published in CACM Volume 63, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Review-article
Popular
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

74
Total Citations
View Citations
16,510
Total Downloads

Downloads (Last 12 months)1,301
Downloads (Last 6 weeks)154

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Kummita SZhang ZBodden EWei SFilkov VRay BZhou M(2024)Visualizing and Understanding the Internals of FuzzingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695284(2199-2204)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695284
Kummita SMiao MBodden EWei SBöhme MNoller YSzekeres L(2024)Visualization Task Taxonomy to Understand the Fuzzing Internals (Registered Report)Proceedings of the 3rd ACM International Fuzzing Workshop10.1145/3678722.3685530(13-22)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3678722.3685530
Yi XLi GHuang WLin XLi JLiu Y(2024)LateBA: Latent Backdoor Attack on Deep Bug Search via Infrequent Execution CodesProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3674806(427-436)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3674806
Arcuri AZhang MGaleotti J(2024)Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIsACM Transactions on Software Engineering and Methodology10.1145/365215733:6(1-36)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3652157
Kraus RNguyen HSchneider M(2024)Generator-based Fuzzing with Input FeaturesProceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing10.1145/3643659.3643925(13-20)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643659.3643925
Predoaia IHarbin JGerasimou SVasiliou CKolovos DGarcía-Domínguez AEgyed AWimmer MChechik MCombemale B(2024)Tree-Based versus Hybrid Graphical-Textual Model Editors: An Empirical Study of Testing SpecificationsProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3640310.3674102(80-91)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3640310.3674102
Sharma STanksalkar SCherupattamoolayil SMachiry AQuek TGao DZhou JCardenas A(2024)Fuzzing API Error Handling Behaviors using Coverage Guided Fault InjectionProceedings of the 19th ACM Asia Conference on Computer and Communications Security10.1145/3634737.3637650(1495-1509)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3634737.3637650
Kumar KMohan CMachiry AVilela JSchulmann HLi N(2024)Precision Guided Approach to Mitigate Data Poisoning Attacks in Federated LearningProceedings of the Fourteenth ACM Conference on Data and Application Security and Privacy10.1145/3626232.3653268(233-244)Online publication date: 19-Jun-2024
https://dl.acm.org/doi/10.1145/3626232.3653268
Xu ZWu BWen CZhang BQin SHe MRoychoudhury APaiva AAbreu RStorey M(2024)RPG: Rust Library Fuzzing with Pool-based Fuzz Target Generation and Generic SupportProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639102(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639102
Debnath JJenkins CSun YChau SChowdhury O(2024)ARMOR: A Formally Verified Implementation of X.509 Certificate Chain Validation2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00220(1462-1480)Online publication date: 19-May-2024
https://doi.org/10.1109/SP54263.2024.00220
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Digital Edition

View this article in digital edition.

Digital Edition

Magazine Site

View this article on the magazine site (external)

Magazine Site

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

Grammar-based whitebox fuzzing

Fuzzing: Brute Force Vulnerability Discovery

Fuzzing for CPS Mutation Testing

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Digital Edition

Magazine Site

Get Access

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations