Resource-aware multi-format network security data storage

Internet security systems like intrusion detection and intrusion prevention systems are based on a simple input-output principle: they receive a high-bandwidth stream of input data and produce summaries of suspicious events. This simple model has serious drawbacks, including the inability to attach context to security alerts, a lack of detailed historical information for anomaly detection baselines, and a lack of detailed forensics information. Together these problems highlight a need for fine-grained security data in the shortterm, and coarse-grained security data in the long-term. To address these limitations we propose resource-aware multi-format security data storage. Our approach is to develop an architecture for recording different granularities of security data simultaneously. To explore this idea we present a novel framework for analyzing security data as a spectrum of information and a set of algorithms for collecting and storing multi-format data. We construct a prototype system and deploy it on darknets at academic, Fortune 100 enterprise, and ISP networks. We demonstrate how a hybrid algorithm that provides guarantees on time and space satisfies the short and long-term goals across a four month deployment period and during a series of large-scale denial of service attacks.