Article

Free access

Estimating flow distributions from sampled flow statistics

Authors:

Mikkel ThorupAuthors Info & Claims

SIGCOMM '03: Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications

Pages 325 - 336

https://doi.org/10.1145/863955.863992

Published: 25 August 2003 Publication History

Abstract

Passive traffic measurement increasingly employs sampling at the packet level. Many high-end routers form flow statistics from a sampled substream of packets. Sampling is necessary in order to control the consumption of resources by the measurement operations. However, knowledge of the statistics of flows in the unsampled stream remains useful, for understanding both characteristics of source traffic, and consumption of resources in the network.This paper provide methods that use flow statistics formed from sampled packet stream to infer the absolute frequencies of lengths of flows in the unsampled stream. A key part of our work is inferring the numbers and lengths of flows of original traffic that evaded sampling altogether. We achieve this through statistical inference, and by exploiting protocol level detail reported in flow records. The method has applications to detection and characterization of network attacks: we show how to estimate, from sampled flow statistics, the number of compromised hosts that are sending attack traffic past the measurement point. We also investigate the impact on our results of different implementations of packet sampling.

References

[1]

J. Apisdorf, K. Claffy, K. Thompson, R. Wilder, "OC3MON: Flexible, Affordable, High Performance Statistics Collection," See: http://www.nlanr.net/NA/Oc3mon

[2]

B.-Y. Choi, J.Park, Zh.-L. Zhang, "Adaptive Random Sampling for Load Change Detection", ACM SIGMETRICS 2002 (Extended Abstract).

Digital Library

[3]

Cisco NetFlow; for further information see http://www.cisco.com/warp/public/732/netflow/index.html

[4]

K. C. Claffy, H.-W. Braun, and G. C. Polyzos. "Parameterizable methodology for internet traffic flow profiling", IEEE Journal on Selected Areas in Communications, vol. 13, no. 8, pp. 1481--1494, Oct. 1995.

Digital Library

[5]

K. C. Claffy, G. C. Polyzos, and H.-W. Braun. "Application of Sampling Methodologies to Network Traffic Characterization", Proceedings ACM SIGCOMM'93, San Francisco, CA, September pp. 13--17, 1993.

Digital Library

[6]

D. Comer, "Internetworking with TCP/IP, Volume 1: Principles, Protocols, and Architecture", Third Edition, Prentice Hall, NJ, 1995.

Digital Library

[7]

A. P. Dempster, N. M. Laird, D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm (with discussion)", J. Roy. Statist. Soc. Ser., vol. 39, pp. 1--38, 1977.

[8]

N. G. Duffield, C. Lund, M. Thorup, "Charging from sampled network usage," ACM SIGCOMM Internet Measurement Workshop 2001, San Francisco, CA, November 1-2, 2001.

Digital Library

[9]

N. G. Duffield, C. Lund, M. Thorup, "Properties and Prediction of Flow Statistics from Sampled Packet Streams", ACM SIGCOMM Internet Measurement Workshop 2002, Marseille, France, November 6-8, 2002.

Digital Library

[10]

C. Estan and G. Varghese, "New Directions in Traffic Measurement and Accounting", Proc SIGCOMM 2002, Pittsburgh, PA, August 19--23, 2002.

Digital Library

[11]

A. Feldmann, R. Caceres, F. Douglis, G. Glass, M. Rabinovich, "Performance of Web Proxy Caching in Heterogeneous Bandwidth Environments," in Proc. IEEE INFOCOM'99, New York, NY, March 23-25, 1999.

[12]

A. Feldmann, J. Rexford, and R. Cáceres, "Efficient Policies for Carrying Web Traffic over Flow-Switched Networks," IEEE/ACM Transactions on Networking, vol. 6, no.6, pp. 673--685, December 1998.

Digital Library

[13]

P.J. Haas and L. Stokes, "Estimating the number of classes in a finite population," J. Amer. Statist. Assoc., vol. 93, pp 1475--1487, 1998.

[14]

Inmon Corporation, "sFlow accuracy and billing", see: http://www.inmon.com/PDF/sFlowBilling.pdf

[15]

P.J. Green, "On the use of the EM algorithm for penalized likelihood estimation," J. R. Statist. Soc. B, vol. 52, pp. 443--452, 1990.

[16]

"Internet Protocol Flow Information eXport" (IPFIX). IETF Working Group. See: http://net.doit.wisc.edu/ipfix/

[17]

D. Moore, V. Paxson, S. Savage, C. Shannon, S. Staniford, N. Weaver, "The Spread of the Sapphire/Slammer Worm", Technical Report, CAIDA, 2003. See http://www.caida.org/outreach/papers/2003/sapphire/sapphire.html.

[18]

NLANR Moat PMA trace archive. See http://pma.nlanr.net/Traces/long/ipls1.html

[19]

V. Paxson, "Empirically-Derived Analytic Models of Wide-Area TCP Connections", IEEE/ACM Transactions on Networking, Vol. 2 No. 4, August 1994.

Digital Library

[20]

V. Paxson, G. Almes, J. Mahdavi, M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998.

Digital Library

[21]

Packet Sampling (PSAMP) IETF Working Group Charter. See http://www.ietf.org/html.charters/psamp-charter.html

[22]

J. Postel, "Transmission Control Protocol," RFC 793, September 1981.

[23]

L. Sachs, "Applied Statistics", Second Edition, Springer, New York, 1984.

[24]

C.F. Jeff Wu, "On the convergence properties of the EM algorithm", Annals of Statistics, vol. 11, pp. 95--103, 1982.

Cited By

Graf JChuprikov PEugster PJahnke P(2024)FARM: Comprehensive Data Center Network Monitoring and Management2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS60910.2024.00055(520-530)Online publication date: 23-Jul-2024
https://doi.org/10.1109/ICDCS60910.2024.00055
Yang KWu YMiao RYang TLiu ZXu ZQiu RZhao YLv HJi ZXie GSchulzrinne HKohler EMaltz DMisra V(2023)ChameleMon: Shifting Measurement Attention as Network State ChangesProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604850(881-903)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604850
Yang KLong SShi QLi YLiu ZWu YYang TJia Z(2023)SketchINT: Empowering INT With TowerSketch for Per-Flow Per-Switch MeasurementIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330392434:11(2876-2894)Online publication date: Nov-2023
https://doi.org/10.1109/TPDS.2023.3303924
Show More Cited By

Index Terms

Estimating flow distributions from sampled flow statistics
1. Mathematics of computing
  1. Probability and statistics
2. Networks
  1. Network services
    1. Network monitoring

Recommendations

Estimating flow distributions from sampled flow statistics

Passive traffic measurement increasingly employs sampling at the packet level. Many high-end routers form flow statistics from a sampled substream of packets. Sampling controls the consumption of resources by the measurement operations. However, ...
Estimating Flow Length Distributions from Double-Sampled Flow Statistics
SCALCOM-EMBEDDEDCOM '09: Proceedings of the 2009 International Conference on Scalable Computing and Communications; Eighth International Conference on Embedded Computing

Knowing the length distributions of traffic flows passing through a network link is useful for some applications such as inferring traffic demands, characterizing source traffic, and detecting traffic anomalies. The collection of the necessary ...
A Novel Method for Estimating Flow Length Distributions from Double-Sampled Flow Statistics
HPCC '10: Proceedings of the 2010 IEEE 12th International Conference on High Performance Computing and Communications

Since the generation of detailed traffic statistics does not scale well with link speed, increasingly passive traffic measurement employs sampling at the packet or flow level. Sampling has become an attractive and scalable means to measure flow data on ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '03: Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications

August 2003

432 pages

ISBN:1581137354

DOI:10.1145/863955

General Chairs:
Anja Feldmann
TU Munich
,
Martina Zitterbart
University of Karlsruhe
,
Program Chairs:
Jon Crowcroft
University of Cambridge
,
David Wetherall
University of Washington

Copyright © 2003 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 August 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGCOMM03

Sponsor:

SIGCOMM03: Applications, Technologies, Architectures, and Protocols for Computer Communications

August 25 - 29, 2003

Karlsruhe, Germany

Acceptance Rates

SIGCOMM '03 Paper Acceptance Rate 34 of 319 submissions, 11%;

Overall Acceptance Rate 462 of 3,389 submissions, 14%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

169
Total Citations
View Citations
1,834
Total Downloads

Downloads (Last 12 months)136
Downloads (Last 6 weeks)31

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Graf JChuprikov PEugster PJahnke P(2024)FARM: Comprehensive Data Center Network Monitoring and Management2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS60910.2024.00055(520-530)Online publication date: 23-Jul-2024
https://doi.org/10.1109/ICDCS60910.2024.00055
Yang KWu YMiao RYang TLiu ZXu ZQiu RZhao YLv HJi ZXie GSchulzrinne HKohler EMaltz DMisra V(2023)ChameleMon: Shifting Measurement Attention as Network State ChangesProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604850(881-903)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604850
Yang KLong SShi QLi YLiu ZWu YYang TJia Z(2023)SketchINT: Empowering INT With TowerSketch for Per-Flow Per-Switch MeasurementIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330392434:11(2876-2894)Online publication date: Nov-2023
https://doi.org/10.1109/TPDS.2023.3303924
Zheng HTian CYang TLin HLiu CZhang ZDou WChen GKuipers FOrda A(2022)FlyMonProceedings of the ACM SIGCOMM 2022 Conference10.1145/3544216.3544239(486-502)Online publication date: 22-Aug-2022
https://dl.acm.org/doi/10.1145/3544216.3544239
Zheng HJiang YTian CCheng LHuang QLi WWang YHuang QZheng JXia RWang YDou WChen G(2022)Rethinking Fine-Grained Measurement From Software-Defined Perspective: A SurveyIEEE Transactions on Services Computing10.1109/TSC.2021.310396815:6(3649-3667)Online publication date: 1-Nov-2022
https://doi.org/10.1109/TSC.2021.3103968
Jia PWang PZhao JTao JYuan YGuan X(2022)Erasable Virtual HyperLogLog for Approximating Cumulative Distribution over Data StreamsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2021.305293834:11(5336-5350)Online publication date: 1-Nov-2022
https://doi.org/10.1109/TKDE.2021.3052938
Wang YWang XXu SHe CZhang YRen JYu S(2022)FlexMonJournal of Network and Computer Applications10.1016/j.jnca.2022.103344201:COnline publication date: 1-May-2022
https://dl.acm.org/doi/10.1016/j.jnca.2022.103344
Sheng SHuang QWang SBao Y(2021)PR-sketchProceedings of the VLDB Endowment10.14778/3467861.346786814:10(1783-1796)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.14778/3467861.3467868
Panda SFeng YKulkarni SRamakrishnan KDuffield NBhuyan LCarle GOtt J(2021)SmartWatchProceedings of the 17th International Conference on emerging Networking EXperiments and Technologies10.1145/3485983.3494861(60-75)Online publication date: 2-Dec-2021
https://dl.acm.org/doi/10.1145/3485983.3494861
Michel OSonchack JCusack GNazari MKeller ESmith J(2021)Software Packet-Level Network Analytics at Cloud ScaleIEEE Transactions on Network and Service Management10.1109/TNSM.2021.305865318:1(597-610)Online publication date: Mar-2021
https://doi.org/10.1109/TNSM.2021.3058653
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents