Article

Impact of packet sampling on anomaly detection metrics

Authors:

Daniela Brauckhoff,

Bernhard Tellenbach,

Anukool LakhinaAuthors Info & Claims

IMC '06: Proceedings of the 6th ACM SIGCOMM conference on Internet measurement

Pages 159 - 164

https://doi.org/10.1145/1177080.1177101

Published: 25 October 2006 Publication History

Abstract

Packet sampling methods such as Cisco's NetFlow are widely employed by large networks to reduce the amount of traffic data measured. A key problem with packet sampling is that it is inherently a lossy process, discarding (potentially useful) information. In this paper, we empirically evaluate the impact of sampling on anomaly detection metrics. Starting with unsampled flow records collected during the Blaster worm outbreak, we reconstruct the underlying packet trace and simulate packet sampling at increasing rates. We then use our knowledge of the Blaster anomaly to build a baseline of normal traffic (without Blaster), against which we can measure the anomaly size at various sampling rates. This approach allows us to evaluate the impact of packet sampling on anomaly detection without being restricted to (or biased by) a particular anomaly detection method.We find that packet sampling does not disturb the anomaly size when measured in volume metrics such as the number of bytes and number of packets, but grossly biases the number of flows. However, we find that recently proposed entropy-based summarizations of packet and flow counts are affected less by sampling, and expose the Blaster worm outbreak even at higher sampling rates. Our findings suggest that entropy summarizations are more resilient to sampling than volume metrics. Thus, while not perfect, sampling still preserves sufficient distributional structure, which when harnessed by tools like entropy, can expose hard-to-detect scanning anomalies.

References

[1]

Barford, P., Kline, J., Plonka, D., and Ron, A. A signal analysis of network traffic anomalies. In Internet Measurement Workshop (Marseille, November 2002).

Digital Library

[2]

Brutlag, J. Aberrant behavior detection in timeseries for network monitoring. In USENIX LISA (New Orleans, December 2000).

Digital Library

[3]

Choi, B.-Y., Park, J., and Zhang, Z.-L. Adaptive random sampling for total load estimation. In IEEE International Conference on Communications (2003).

Digital Library

[4]

Cisco NetFlow. At www.cisco.com/warp/public/732/Tech/netflow/.

[5]

Duffield, N., Lund, C., and Thorup, M. Properties and prediction of flow statistics from sampled packet streams. In ACM SIGCOMM Internet Measurment Workshop (2002).

Digital Library

[6]

Duffield, N., Lund, C., and Thorup, M. Estimating Flow Distributions from Sampled Flow Statistics. In ACM SIGCOMM (Karlsruhe, August 2003).

Digital Library

[7]

Estan, C., and Varghese, G. New directions in traffic measurement and accounting. In Proceedings of the 2001 ACM SIGCOMM Internet Measurement Workshop (San Francisco, CA, 2001), pp. 75--80.

Digital Library

[8]

Hohn, N., and Veitch, D. Inverting Sampled Traffic. Internet Measurement Conference (Miami, October 2003).

Digital Library

[9]

Jung, J., Krishnamurthy, B., and Rabinovich, M. Flash crowds and denial of service attacks: Characterization and implications for cdns and web sites. In Proceedings of the International World Wide Web Conference (2002).

Digital Library

[10]

Jung, J., Paxson, V., Berger, A., and Balakrishnan, H. Fast portscan detection using sequential hypothesis testing. In Proceedings of the IEEE Symposium on Security and Privacy (2004).

[11]

Kim, M.-S., Kang, H.-J., Hung, S.-C., Chung, S.-H., and Hong, J. W. A Flow-based Method for Abnormal Network Traffic Detection. IEEE/IFIP Network Operations and Management Symposium (Seoul, 2004).

[12]

Lakhina, A., Crovella, M., and Diot, C. Diagnosing Network-Wide Traffic Anomalies. ACM SIGCOMM (Portland, August 2004).

Digital Library

[13]

Lakhina, A., Crovella, M., and Diot, C. Mining Anomalies Using Traffic Feature Distributions. ACM SIGCOMM (Philadelphia, August 2005).

Digital Library

[14]

Mai, J., Chuah, C.-N., Sridharan, A., Ye, T., and Zang, H. Is sampled data sufficient for anomaly detection? IMC 2006 (Rio de Janeiro, Brazil, October 2006).

Digital Library

[15]

Mai, J., Sridharan, A., Chuah, C.-N., Zang, H., and Ye, T. Impact of packet sampling on portscan detection. IEEE Journal on Selected Areas in Communication (2006).

[16]

Müller, O., Graf, D., Oppermann, A., and Weibel, H. Swiss internet analysis, 2004. http://www.swiss-internet-analysis.org/.

[17]

Sridharan, A., Ye, T., and Bhattacharrya, S. Connectionless port scan detection on the backbone. Malware workshop, held in conjunction with IPCCC (Phoenix, AZ, April 2006).

[18]

SWITCH. Swiss academic and research network. http://www.switch.ch/, 2006.

[19]

Wagner, A., and Plattner, B. Entropy based worm and anomaly detection in fast ip networks. In Proceedings of the STCA security workshop / WETICE 2005 (2005).

Digital Library

[20]

Wallerich, J., Dreger, H., Feldmann, A., Krishnamurthy, B., and Willinger, W. A methodology for studying persistency aspects of internet flows. SIGCOMM Comput. Commun. Rev. 35, 2 (2005).

Digital Library

[21]

Xu, K., Zhang, Z.-L., and Bhattacharrya, S. Profiling internet backbone traffic: Behavior models and applications. In ACM Sigcomm 2005 (Philadelphia, PA, August 2005).

Digital Library

Cited By

Kumar GLagesse B(2024)Thimblerig: A Game-Theoretic, Adaptive, Risk-limiting Security System for Cloud SystemsNOMS 2024-2024 IEEE Network Operations and Management Symposium10.1109/NOMS59830.2024.10575857(1-6)Online publication date: 6-May-2024
https://doi.org/10.1109/NOMS59830.2024.10575857
Hajj SAzar JBou Abdo JDemerjian JGuyeux CMakhoul AGinhac D(2023)Cross-Layer Federated Learning for Lightweight IoT Intrusion Detection SystemsSensors10.3390/s2316703823:16(7038)Online publication date: 9-Aug-2023
https://doi.org/10.3390/s23167038
Clausen HManocha AGibson M(2022)Detecting Proxies Relaying Streaming Internet TrafficIEEE INFOCOM 2022 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)10.1109/INFOCOMWKSHPS54753.2022.9797963(1-6)Online publication date: 2-May-2022
https://doi.org/10.1109/INFOCOMWKSHPS54753.2022.9797963
Show More Cited By

Index Terms

Impact of packet sampling on anomaly detection metrics
1. Networks
  1. Network components
  2. Network types
    1. Public Internet

Recommendations

Is sampled data sufficient for anomaly detection?
IMC '06: Proceedings of the 6th ACM SIGCOMM conference on Internet measurement

Sampling techniques are widely used for traffic measurements at high link speed to conserve router resources. Traditionally, sampled traffic data is used for network management tasks such as traffic matrix estimations, but recently it has also been used ...
CNN-based anomaly detection for packet payloads of industrial control system

Industrial control systems (ICSs) are more vulnerable to cyber threats owing to their network connectivity. The intrusion detection system(IDS) has been deployed to detect sophisticated cyber-attack but the existing IDS uses the packet header information ...
On mitigating sampling-induced accuracy loss in traffic anomaly detection systems

Real-time Anomaly Detection Systems (ADSs) use packet sampling to realize traffic analysis at wire speeds. While recent studies have shown that a considerable loss of anomaly detection accuracy is incurred due to sampling, solutions to mitigate this ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IMC '06: Proceedings of the 6th ACM SIGCOMM conference on Internet measurement

October 2006

356 pages

ISBN:1595935614

DOI:10.1145/1177080

General Chairs:
Jussara Almeida
Federal University of Minas Gerais, Brazil
,
Virgilio Almeida
Federal University of Minas Gerais, Brazil
,
Program Chair:
Paul Barford
University of Wisconsin -- Madison, USA

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 October 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

IMC06

Sponsor:

IMC06: Internet Measurement Conference

October 25 - 27, 2006

Rio de Janeriro, Brazil

Acceptance Rates

Overall Acceptance Rate 277 of 1,083 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

152
Total Citations
View Citations
1,197
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)3

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kumar GLagesse B(2024)Thimblerig: A Game-Theoretic, Adaptive, Risk-limiting Security System for Cloud SystemsNOMS 2024-2024 IEEE Network Operations and Management Symposium10.1109/NOMS59830.2024.10575857(1-6)Online publication date: 6-May-2024
https://doi.org/10.1109/NOMS59830.2024.10575857
Hajj SAzar JBou Abdo JDemerjian JGuyeux CMakhoul AGinhac D(2023)Cross-Layer Federated Learning for Lightweight IoT Intrusion Detection SystemsSensors10.3390/s2316703823:16(7038)Online publication date: 9-Aug-2023
https://doi.org/10.3390/s23167038
Clausen HManocha AGibson M(2022)Detecting Proxies Relaying Streaming Internet TrafficIEEE INFOCOM 2022 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)10.1109/INFOCOMWKSHPS54753.2022.9797963(1-6)Online publication date: 2-May-2022
https://doi.org/10.1109/INFOCOMWKSHPS54753.2022.9797963
Lai YTsai CChuang CKu XChen J(2022)Tabular Interpolation Approach Based on Stable Random Projection for Estimating Empirical Entropy of High-Speed Network TrafficIEEE Access10.1109/ACCESS.2022.321033610(104934-104953)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3210336
Novo CSilva JMorla R(2021)An Outlook on using Packet Sampling in Flow-based C2 TLS Malware Traffic Detection2021 12th International Conference on Network of the Future (NoF)10.1109/NoF52522.2021.9609889(1-5)Online publication date: 6-Oct-2021
https://doi.org/10.1109/NoF52522.2021.9609889
Hajj SSibai RAbdo JDemerjian JGuyeux CMakhoul AGinhac D(2021)A Critical Review on the Implementation of Static Data Sampling Techniques to Detect Network AttacksIEEE Access10.1109/ACCESS.2021.31186059(138903-138938)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3118605
Polverini MCianfrani AListanti M(2020)A Theoretical Framework for Network Monitoring Exploiting Segment Routing CountersIEEE Transactions on Network and Service Management10.1109/TNSM.2020.300180917:3(1924-1940)Online publication date: Sep-2020
https://doi.org/10.1109/TNSM.2020.3001809
Qin TLiu ZWang PLi SGuan XGao L(2020)Symmetry Degree Measurement and its Applications to Anomaly DetectionIEEE Transactions on Information Forensics and Security10.1109/TIFS.2019.293373115(1040-1055)Online publication date: 2020
https://doi.org/10.1109/TIFS.2019.2933731
Sibai RAbdo JChabchoub YDemerjian JChiky RBarbar K(2020)Data Summarization Using Sampling Algorithms: Data Stream Case StudyPrinciples of Data Science10.1007/978-3-030-43981-1_6(105-124)Online publication date: 9-Jul-2020
https://doi.org/10.1007/978-3-030-43981-1_6
Zhao KWang JQi HXie XZhou XLi K(2020)HBL-Sketch: A New Three-Tier Sketch for Accurate Network MeasurementAlgorithms and Architectures for Parallel Processing10.1007/978-3-030-38991-8_4(48-59)Online publication date: 22-Jan-2020
https://doi.org/10.1007/978-3-030-38991-8_4
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents