FineMon: An Innovative Adaptive Network Telemetry Scheme for Fine-Grained, Multi-Metric Data Monitoring with Dynamic Frequency Adjustment and Enhanced Data Recovery
Article No.: 12, Pages 1 - 26
Abstract
Network telemetry, characterized by its efficient push model and high-performance communication protocol (gRPC), offers a new avenue for collecting fine-grained real-time data. Despite its advantages, existing network telemetry systems lack a theoretical basis for setting measurement frequency, struggle to capture informative samples, and face challenges in setting a uniform frequency for multi-metric monitoring. We introduce FineMon, an innovative adaptive network telemetry scheme for precise, fine-grained, multi-metric data monitoring. FineMon leverages a novel Two-sided Frequency Adjustment (TFA) to dynamically adjust the measurement frequency on the Network Management System (NMS) and infrastructure sides. On the NMS side, we provide a theoretical basis for frequency determination, drawing on changes in the rank of multi-metric data to minimize monitoring overhead. On the infrastructure side, we adjust the frequency in real-time to capture significant data fluctuations. We propose a robust Enhanced-Subspace-based Tensor Completion (ESTC) to ensure accurate recovery of fine-grained data, even with noise or outliers. Through extensive experimentation with three real datasets, we demonstrate FineMon's superiority over existing schemes in reduced measurement overhead, enhanced accuracy, and effective capture of intricate temporal features.
Supplemental Material
MP4 File
Presentation video
- Download
- 70.06 MB
MP4 File
presentation video
- Download
- 73.18 MB
References
[1]
Hiroshi Akima. 1970. A new method of interpolation and smooth curve fitting based on local procedures. Journal of the ACM (JACM) 17, 4 (1970), 589--602.
[2]
Hiroshi Akima. 1974. A method of bivariate interpolation and smooth surface fitting based on local procedures. Commun. ACM 17, 1 (1974), 18--20.
[3]
Maria-Florina F Balcan and Hongyang Zhang. 2016. Noise-tolerant life-long matrix completion via adaptive sampling. Advances in Neural Information Processing Systems 29 (2016).
[4]
Thierry Blu, Philippe Thévenaz, and Michael Unser. 2004. Linear interpolation revitalized. IEEE Transactions on Image Processing 13, 5 (2004), 710--719.
[5]
CAIDA. 2018. The CAIDA UCSD Anonymized Internet Traces. [Online]. Available: https://www.caida.org/catalog/datasets/passive_dataset.
[6]
Emmanuel Candes and Benjamin Recht. 2012. Exact matrix completion via convex optimization. Commun. ACM 55, 6 (2012), 111--119.
[7]
Emmanuel J Candès, Justin Romberg, and Terence Tao. 2006. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on information theory 52, 2 (2006), 489--509.
[8]
Emmanuel J Candès and Terence Tao. 2010. The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory 56, 5 (2010), 2053--2080.
[9]
Wei Cao, Yusong Gao, Feifei Li, Sheng Wang, Bingchen Lin, Ke Xu, Xiaojie Feng, Yucong Wang, Zhenjun Liu, and Gejin Zhang. 2020. Timon: A timestamped event database for efficient telemetry data processing and analytics. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 739--753.
[10]
Jeffrey D Case, Mark Fedor, Martin L Schoffstall, and James Davin. 1989. Simple network management protocol (SNMP). Technical Report.
[11]
Kenjiro Cho, Koushirou Mitsuya, and Akira Kato. 2000. Traffic data repository at the {WIDE} project. In 2000 USENIX Annual Technical Conference (USENIX ATC 00).
[12]
Lei Deng, Xiao-Yang Liu, Haifeng Zheng, Xinxin Feng, and Youjia Chen. 2021. Graph spectral regularized tensor completion for traffic data imputation. IEEE Transactions on Intelligent Transportation Systems 23, 8 (2021), 10996--11010.
[13]
Lei Deng, Haifeng Zheng, Xiao-Yang Liu, Xinxin Feng, and Zhizhang David Chen. 2020. Network latency estimation with leverage sampling for personal devices: An adaptive tensor completion approach. IEEE/ACM Transactions on Networking 28, 6 (2020), 2797--2808.
[14]
Frederick N Fritsch and Ralph E Carlson. 1980. Monotone piecewise cubic interpolation. SIAM J. Numer. Anal. 17, 2 (1980), 238--246.
[15]
Hancheng Ge, Kai Zhang, Majid Alfifi, Xia Hu, and James Caverlee. 2018. DisTenC: A distributed algorithm for scalable tensor completion on spark. In 2018 IEEE 34th International Conference on Data Engineering (ICDE). IEEE, 137--148.
[16]
Google. 2015. gRPC. https://grpc.io/.
[17]
Arpit Gupta, Rob Harrison, Marco Canini, Nick Feamster, Jennifer Rexford, and Walter Willinger. 2018. Sonata: Query-driven streaming network telemetry. In Proceedings of the 2018 conference of the ACM special interest group on data communication. 357--371.
[18]
Richard A Harshman et al. 1970. Foundations of the PARAFAC procedure: Models and conditions for an" explanatory" multimodal factor analysis. (1970).
[19]
Guanghui He and Jennifer C Hou. 2006. On sampling self-similar Internet traffic. Computer Networks 50, 16 (2006), 2919--2936.
[20]
Zilong He, Pengfei Chen, Xiaoyun Li, Yongfeng Wang, Guangba Yu, Cailin Chen, Xinrui Li, and Zibin Zheng. 2020. A spatiotemporal deep learning approach for unsupervised anomaly detection in cloud systems. IEEE Transactions on Neural Networks and Learning Systems (2020).
[21]
Rick Hofstede, Pavel Celeda, Brian Trammell, Idilio Drago, Ramin Sadre, Anna Sperotto, and Aiko Pras. 2014. Flow monitoring explained: From packet capture to data analysis with netflow and ipfix. IEEE Communications Surveys & Tutorials 16, 4 (2014), 2037--2064.
[22]
Bo Huang, Cun Mu, Donald Goldfarb, and John Wright. 2014. Provable low-rank tensor recovery. Optimization-Online 4252, 2 (2014), 455--500.
[23]
Bo Hui, Da Yan, Haiquan Chen, and Wei-Shinn Ku. 2022. Time-sensitive POI Recommendation by Tensor Completion with Side Information. In 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 205--217.
[24]
Liuwei Huo, Dingde Jiang, Xiangnan Zhu, and Huibin Jia. 2019. An Adaptive Measurement Method for Flow Traffic in Software Defined Networking. In Simulation Tools and Techniques: 11th International Conference, SIMUtools 2019, Chengdu, China, July 8--10, 2019, Proceedings 11. Springer, 115--124.
[25]
Zoom Video Communications Inc. 2020. Zoom Client Connection Process Whitepaper. https://explore.zoom.us/docs/doc/Zoom_Client_Connection%20Process_Whitepaper.pdf.
[26]
Yuting Jiang, Yifan Xiong, Lei Qu, Cheng Luo, Chen Tian, Peng Cheng, and Yongqiang Xiong. 2022. Moneo: Non-intrusive Fine-grained Monitor for AI Infrastructure. In ICC 2022-IEEE International Conference on Communications. IEEE, 2586--2591.
[27]
Misha E Kilmer, Karen Braman, Ning Hao, and Randy C Hoover. 2013. Third-order tensors as operators on matrices: A theoretical and computational framework with applications in imaging. SIAM J. Matrix Anal. Appl. 34, 1 (2013), 148--172.
[28]
Akshay Krishnamurthy and Aarti Singh. 2013. Low-rank matrix and tensor completion via adaptive sampling. Advances in neural information processing systems 26 (2013).
[29]
Akshay Krishnamurthy and Aarti Singh. 2014. On the power of adaptivity in matrix completion and approximation. arXiv preprint arXiv:1407.3619 (2014).
[30]
Sue E Leurgans, Robert T Ross, and Rebecca B Abel. 1993. A decomposition for three-way arrays. SIAM J. Matrix Anal. Appl. 14, 4 (1993), 1064--1083.
[31]
Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, et al. 2019. HPCC: High precision congestion control. In Proceedings of the ACM Special Interest Group on Data Communication. 44--58.
[32]
Ji Liu, Przemyslaw Musialski, Peter Wonka, and Jieping Ye. 2012. Tensor completion for estimating missing values in visual data. IEEE transactions on pattern analysis and machine intelligence 35, 1 (2012), 208--220.
[33]
Xiao-Yang Liu, Shuchin Aeron, Vaneet Aggarwal, Xiaodong Wang, and Min-You Wu. 2015. Adaptive sampling of RF fingerprints for fine-grained indoor localization. IEEE Transactions on Mobile Computing 15, 10 (2015), 2411--2423.
[34]
Yipeng Liu, Zhen Long, Huyan Huang, and Ce Zhu. 2019. Low CP rank and tucker rank tensor completion for estimating missing components in image data. IEEE Transactions on Circuits and Systems for Video Technology 30, 4 (2019), 944--954.
[35]
Yuanyuan Liu, Fanhua Shang, Wei Fan, James Cheng, and Hong Cheng. 2015. Generalized higher order orthogonal iteration for tensor learning and decomposition. IEEE transactions on neural networks and learning systems 27, 12 (2015), 2551--2563.
[36]
Yuanyuan Liu, Fanhua Shang, Licheng Jiao, James Cheng, and Hong Cheng. 2014. Trace norm regularized CANDE-COMP/PARAFAC decomposition with missing data. IEEE transactions on cybernetics 45, 11 (2014), 2437--2448.
[37]
Nick McKeown, Tom Anderson, Hari Balakrishnan, Guru Parulkar, Larry Peterson, Jennifer Rexford, Scott Shenker, and Jonathan Turner. 2008. OpenFlow: enabling innovation in campus networks. ACM SIGCOMM computer communication review 38, 2 (2008), 69--74.
[38]
Sky McKinley and Megan Levine. 1998. Cubic spline interpolation. College of the Redwoods 45, 1 (1998), 1049--1060.
[39]
Xiaonan Nie, Xupeng Miao, Zhi Yang, and Bin Cui. 2022. Tsplit: Fine-grained gpu memory management for efficient dnn training via tensor splitting. In 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 2615--2628.
[40]
Bernardino Romera-Paredes and Massimiliano Pontil. 2013. A new convex relaxation for tensor completion. Advances in neural information processing systems 26 (2013).
[41]
Josep Sanjuas-Cuxart, Pere Barlet-Ros, Nick Duffield, and Ramana Kompella. 2012. Cuckoo sampling: Robust collection of flow aggregates under a fixed memory budget. In 2012 Proceedings IEEE INFOCOM. IEEE, 2751--2755.
[42]
Robin Sommer and Anja Feldmann. 2002. NetFlow: Information loss or win?. In Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment. 173--174.
[43]
Ryota Tomioka, Kohei Hayashi, and Hisashi Kashima. 2010. Estimation of low-rank tensors via convex optimization. arXiv preprint arXiv:1010.0789 (2010).
[44]
Ledyard R Tucker. 1966. Some mathematical notes on three-mode factor analysis. Psychometrika 31, 3 (1966), 279--311.
[45]
Mea Wang, Baochun Li, and Zongpeng Li. 2004. sFlow: Towards resource-efficient and agile service federation in service overlay networks. In 24th International Conference on Distributed Computing Systems, 2004. Proceedings. IEEE, 628--635.
[46]
Canhua Wu, Nengwen Zhao, Lixin Wang, Xiaoqin Yang, Shining Li, Ming Zhang, Xing Jin, Xidao Wen, Xiaohui Nie, Wenchi Zhang, et al . 2021. Identifying root-cause metrics for incident diagnosis in online service systems. In 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE). IEEE, 91--102.
[47]
Qin Wu, John Strassner, Adrian Farrel, and Liang Zhang. 2016. Network Telemetry and Big Data Analysis. Internet-Draft draft-wu-t2trg-network-telemetry-00. Internet Engineering Task Force. https://datatracker.ietf.org/doc/draft-wu-t2trg-network-telemetry/00/ Work in Progress.
[48]
Kun Xie, Yuxiang Chen, Xin Wang, Gaogang Xie, Jiannong Cao, Jigang Wen, Guangming Yang, and Jiaqi Sun. 2020. Accurate and fast recovery of network monitoring data with GPU-accelerated tensor completion. IEEE/ACM Transactions on Networking 28, 4 (2020), 1601--1614.
[49]
Yangyang Xu and Wotao Yin. 2013. A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM Journal on imaging sciences 6, 3 (2013), 1758--1789.
[50]
Zemin Zhang and Shuchin Aeron. 2016. Exact tensor completion using t-SVD. IEEE Transactions on Signal Processing 65, 6 (2016), 1511--1526.
Index Terms
- FineMon: An Innovative Adaptive Network Telemetry Scheme for Fine-Grained, Multi-Metric Data Monitoring with Dynamic Frequency Adjustment and Enhanced Data Recovery
Recommendations
A fine blind frequency offset estimator for OFDM/OQAM systems
Like other orthogonal frequency division multiplexing (OFDM) systems, OFDM systems based on offset quadrature amplitude modulation (OFDM/OQAM) are very sensitive to carrier frequency offset. In this paper, a new blind carrier frequency offset estimator ...
NT-RP: A High-Versatility Approach for Network Telemetry Based on FPGA Dynamic Reconfigurable Pipeline
Wireless Algorithms, Systems, and ApplicationsAbstractNetwork telemetry provides more accurate and reliable services for intelligent network control by pushing fresh status information actively with help of data plane. However, most existing network telemetry methods are difficult to be deployed ...
Real‐time and fine‐grained network monitoring using in‐band network telemetry
SummaryIn the modern era of software‐defined networking (SDN), network monitoring is becoming more important for providing information about a network and helping SDN controllers to make decisions about the network. In‐band Network Telemetry (INT) is a ...
In this paper, we present the design of the overall INT management architecture and its two main components: the INT management system which controls heterogeneous INT‐capable devices through a common interface, and INTCollector, a high‐performance ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
February 2024
1874 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 March 2024
Published in PACMMOD Volume 2, Issue 1
Author Tags
Qualifiers
- Research-article
Funding Sources
- Key Research and Development Program of Hunan Province of China
- National Science Foundation for Distinguished Young Scholars
- National Natural Science Foundation of China-Research Grants Council (RGC) Joint Research Scheme
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 135Total Downloads
- Downloads (Last 12 months)135
- Downloads (Last 6 weeks)17
Reflects downloads up to 22 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in