Abstract
The last twenty years have witnessed the constant reaction of the security community to memory corruption attacks and the evolution of attacking techniques in order to circumvent the newly-deployed countermeasures. In this evolution, the heap of a process received little attention and thus today, the problem of heap overflows is largely unsolved.
In this paper we present HeapSentry, a system designed to detect and stop heap overflow attacks through the cooperation of the memory allocation library of a program and the operating system’s kernel. HeapSentry places unique random canaries at the end of each heap object which are later checked by the kernel, before system calls are allowed to proceed. HeapSentry operates on binaries (no source code needed) and has, by design, no false-positives. At the same time, the active involvement of the kernel provides stronger security guarantees than the current state of the art in heap protection mechanisms for a modest performance overhead.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Adobe: Security bulletins and advisories, http://www.adobe.com/support/security/
Akritidis, P., Costa, M., Castro, M., Hand, S.: Baggy bounds checking: An efficient and backwards-compatible defense against out-of-bounds errors. In: Proceedings of the 18th USENIX Security Symposium, Montreal, QC (August 2009)
Aleph1: Smashing the stack for fun and profit. Phrack, 49 (1996)
Anley, C., Heasman, J., Linder, F.F., Richarte, G.: The Shellcoder’s Handbook: Discovering and Exploiting Security Holes, 2nd edn. Wiley Publishing (2007)
Berger, E.D.: Heapshield: Library-based heap overflow protection for free. UMass CS TR 06-28 (2006)
Berger, E.D., Zorn, B.G.: Diehard: Probabilistic memory safety for unsafe languages. In: Proceedings of 27th Conference on Programming Language Design and Implementation (June 2006)
Bernaschi, M., Gabrielli, E., Mancini, L.V.: Operating system enhancements to prevent the misuse of system calls. In: Proceedings of the 7th Conference on Computer and Communications Security (2000)
Bhatkar, S., DuVarney, D.C., Sekar, R.: Address obfuscation: An efficient approach to combat a broad range of memory error exploits. In: Proceedings of the 12th USENIX Security Symposium, Washington, D.C., pp. 105–120 (August 2003)
Bhatkar, S., Sekar, R.: Data space randomization. In: Zamboni, D. (ed.) DIMVA 2008. LNCS, vol. 5137, pp. 1–22. Springer, Heidelberg (2008)
Checkoway, S., Davi, L., Dmitrienko, A., Sadeghi, A.-R., Shacham, H., Winandy, M.: Return-oriented programming without returns. In: Proceedings of CCS 2010. ACM Press (2010)
Chen, S., Xu, J., Sezer, E.C., Gauriar, P., Iyer, R.K.: Non-control-data attacks are realistic threats. In: Proceedings of the 14th USENIX Security Symposium, Baltimore, MD (August 2005)
Conover, M.: w00w00 on heap overflows, http://www.w00w00.org/files/articles/heaptut.txt
Cowan, C., Pu, C., Maier, D., Hinton, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., Zhang, Q.: StackGuard: Automatic adaptive detection and prevention of buffer-overflow attacks. In: Proceedings of the 7th USENIX Security Symposium (1998)
Designer, S.: lpr LIBC RETURN exploit, http://insecure.org/sploits/linux.libc.return.lpr.sploit.html
Dhurjati, D., Adve, V.: Backwards-compatible array bounds checking for C with very low overhead. In: Proceeding of the 28th International Conference on Software Engineering, Shanghai, China (2006)
Egele, M., Wurzinger, P., Kruegel, C., Kirda, E.: Defending browsers against drive-by downloads: Mitigating heap-spraying code injection attacks. In: Flegel, U., Bruschi, D. (eds.) DIMVA 2009. LNCS, vol. 5587, pp. 88–106. Springer, Heidelberg (2009)
Gal, A., Eich, B., Shaver, M., Anderson, D., Mandelin, D., Haghighat, M.R., Kaplan, B., Hoare, G., Zbarsky, B., Orendorff, J., Ruderman, J., Smith, E.W., Reitmaier, R., Bebenita, M., Chang, M., Franz, M.: Trace-based just-in-time type specialization for dynamic languages. In: ACM Conference on Programming Language Design and Implementation (2009)
IBM: Gcc extension for protecting applications from stack-smashing attacks, http://www.trl.ibm.com/projects/security/ssp/
Jones, R.W.M., Kelly, P.H.J.: Backwards-compatible bounds checking for arrays and pointers in C programs. In: Proceedings of the 3rd International Workshop on Automatic Debugging, Linköping, Sweden, pp. 13–26 (1997)
Kc, G.S., Keromytis, A.D.: e-NeXSh: Achieving an effectively non-executable stack and heap via system-call policing. In: Annual Computer Security Applictions Conference (2005)
Keniston, J., Panchamukhi, P.S., Hiramatsu, M.: Kernel probes (kprobes)
Lin, C., Rajagopalan, M., Baker, S., Collberg, C., Debray, S., Hartman, J.: Protecting against unexpected system calls. In: Proceedings of the 14th USENIX Security Symposium, Baltimore, Maryland. USENIX Association (August 2005)
Lvin, V.B., Novark, G., Berger, E.D., Zorn, B.G.: Archipelago: trading address space for reliability and security. In: Proceedings of the 13th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XIII. ACM (2008)
Microsoft: Security advisories, http://www.microsoft.com/technet/security/advisory/
Novark, G., Berger, E.D.: Dieharder: securing the heap. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, CCS 2010, pp. 573–584. ACM, New York (2010)
National Vulnerability Database, http://nvd.nist.gov
PaX: Documentation for the PaX project, http://pax.grsecurity.net/
Payer, M.: I control your code. In: Proceedings of the 27th Chaos Communication Congress (27c3) (2010)
Provos, N.: Improving host security with system call policies. In: Proceedings of the 12th USENIX Security Symposium, Washington, D.C. (August 2003)
Rivner, U.: Anatomy of the rsa attack, http://blogs.rsa.com/rivner/anatomy-of-an-attack/
Robertson, W., Kruegel, C., Mutz, D., Valeur, F.: Run-time detection of heap-based overflows. In: Proceedings of the 17th Large Installation Systems Administrators Conference, San Diego, CA, pp. 51–60 (October 2003)
Roglia, G.F., Martignoni, L., Paleari, R., Bruschi, D.: Surgically returning to randomized lib(c). In: 25th Annual Computer Security Applications Conference (2009)
Shacham, H.: The geometry of innocent flesh on the bone: Return-into-libc without function calls (on the x86). In: Proceedings of the 14th ACM Conference on Computer and Communications Security (2007)
Solar Designer: Non-executable user stack, http://www.openwall.com/linux/
Spafford, E.H.: The internet worm program: An analysis. Computer Communication Review 19 (1988)
Strace(1): trace system calls/signals, http://linux.die.net/man/1/strace
Van Acker, S., Nikiforakis, N., Philippaerts, P., Younan, Y., Piessens, F.: ValueGuard: Protection of Native Applications against Data-Only Buffer Overflows. In: Jha, S., Mathuria, A. (eds.) ICISS 2010. LNCS, vol. 6503, pp. 156–170. Springer, Heidelberg (2010)
Wilander, J., Nikiforakis, N., Younan, Y., Kamkar, M., Joosen, W.: Ripe: Runtime intrusion prevention evaluator. In: Proceedings of the 27th Annual Computer Security Applications Conference, ACSAC (2011)
Younan, Y., Joosen, W., Piessens, F.: Efficient protection against heap-based buffer overflows without resorting to magic. In: Proceedings of the International Conference on Information and Communication Security, Raleigh, NC (December 2006)
Younan, Y., Joosen, W., Piessens, F.: Runtime countermeasures for code injection attacks against C and C++ programs. ACM Computing Surveys 44(3), 17:1–17:28 (2012)
Zeng, Q., Wu, D., Liu, P.: Cruiser: concurrent heap buffer overflow monitoring using lock-free data structures. In: Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Nikiforakis, N., Piessens, F., Joosen, W. (2013). HeapSentry: Kernel-Assisted Protection against Heap Overflows. In: Rieck, K., Stewin, P., Seifert, JP. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2013. Lecture Notes in Computer Science, vol 7967. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39235-1_11
Download citation
DOI: https://doi.org/10.1007/978-3-642-39235-1_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39234-4
Online ISBN: 978-3-642-39235-1
eBook Packages: Computer ScienceComputer Science (R0)