End-to-End Performance Analysis of Learning-enabled Systems
Authors: 
Pooria Namyar, Michael Schapira, 
Ramesh Govindan, Santiago Segarra, Ryan Beckett, Siva Kesava Reddy Kakarla, Behnaz ArzaniAuthors Info & Claims
Pages 86 - 94
Abstract
We propose a performance analysis tool for learning-enabled systems that allows operators to uncover potential performance issues before deploying DNNs in their systems. The tools that exist for this purpose require operators to faithfully model all components (a white-box approach) or do inefficient black-box local search. We propose a gray-box alternative, which eliminates the need to precisely model all the system's components. Our approach is faster and finds substantially worse scenarios compared to prior work. We show that a state-of-the-art learning-enabled traffic engineering pipeline can underperform the optimal by 6× --- a much higher number compared to what the authors found.
References
[1]
Firas Abuzaid, Srikanth Kandula, Behnaz Arzani, Ishai Menache, Matei Zaharia, and Peter Bailis. 2021. Contracting Wide-area Network Topologies to Solve Flow Problems Quickly. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). USENIX Association, 175--200. https://www.usenix.org/conference/nsdi21/presentation/abuzaid
[2]
Anup Agarwal, Venkat Arun, Devdeep Ray, Ruben Martins, and Srinivasan Seshan. 2024. Towards provably performant congestion control. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, Santa Clara, CA, 951--978. https://www.usenix.org/conference/nsdi24/presentation/agarwal-anup
[3]
Abd AlRhman AlQiam, Yuanjun Yao, Zhaodong Wang, Satyajeet Singh Ahuja, Ying Zhang, Sanjay G. Rao, Bruno Ribeiro, and Mohit Tawarmalani. 2024. Transferable Neural WAN TE for Changing Topologies. In Proceedings of the ACM SIGCOMM 2024 Conference.
[4]
Mina Tahmasbi Arashloo, Ryan Beckett, and Rachit Agarwal. 2023. Formal Methods for Network Performance Analysis. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 645--661. https://www.usenix.org/conference/nsdi23/presentation/tahmasbi
[5]
Venkat Arun, Mina Tahmasbi Arashloo, Ahmed Saeed, Mohammad Alizadeh, and Hari Balakrishnan. [n. d.]. Toward formally verifying congestion control behavior. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference (SIGCOMM '21).
[6]
Yossi Azar, Edith Cohen, Amos Fiat, Haim Kaplan, and Harald Racke. 2003. Optimal oblivious routing in polynomial time. In Proceedings of the Thirty-Fifth Annual ACM Symposium on Theory of Computing (STOC '03). Association for Computing Machinery.
[7]
Stephen Boyd and Lieven Vandenberghe. 2004. Convex optimization. Cambridge university press.
[8]
Lawrence Davis. 1991. Bit-Climbing, Representational Bias, and Test Suite Design. In Proceedings of the 4th International Conference on Genetic Algorithms, Richard K. Belew and Lashon B. Booker (Eds.).
[9]
Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: an efficient SMT solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (Budapest, Hungary) (TACAS'08/ETAPS'08). Springer-Verlag, Berlin, Heidelberg, 337--340.
[10]
Tomer Eliyahu, Yafim Kazak, Guy Katz, and Michael Schapira. 2021. Verifying learning-augmented systems. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference (Virtual Event, USA) (SIGCOMM '21). Association for Computing Machinery, New York, NY, USA, 305--318. https://doi.org/10.1145/3452296.3472936
[11]
Matteo Fischetti and Jason Jo. 2018. Deep neural networks and mixed integer linear optimization. Constraints 23, 3 (2018), 296--309.
[12]
Tomer Gilad, Nathan H. Jay, Michael Shnaiderman, Brighten Godfrey, and Michael Schapira. 2019. Robustifying Network Protocols with Adversarial Examples. In Proceedings of the 18th ACM Workshop on Hot Topics in Networks (HotNets '19).
[13]
Saksham Goel, Benjamin Mikek, Jehad Aly, Venkat Arun, Ahmed Saeed, and Aditya Akella. 2023. A Performance Verification Methodology for Resource Allocation Heuristics. https://api.semanticscholar.org/CorpusID:255595504
[14]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT press.
[15]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems 27 (2014).
[16]
Gurobi Optimization, LLC. 2023. Gurobi Optimizer Reference Manual. https://www.gurobi.com
[17]
Chi-Yao Hong, Srikanth Kandula, Ratul Mahajan, Ming Zhang, Vijay Gill, Mohan Nanduri, and Roger Wattenhofer. 2013. Achieving high utilization with software-driven WAN. SIGCOMM Comput. Commun. Rev. 43, 4 (aug 2013), 15--26. https://doi.org/10.1145/2534169.2486012
[18]
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. 2017. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284 (2017).
[19]
Sushant Jain, Alok Kumar, Subhasree Mandal, Joon Ong, Leon Poutievski, Arjun Singh, Subbaiah Venkata, Jim Wanderer, Junlan Zhou, Min Zhu, Jon Zolla, Urs Hölzle, Stephen Stuart, and Amin Vahdat. 2013. B4: experience with a globally-deployed software defined wan. In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM (Hong Kong, China) (SIGCOMM '13). Association for Computing Machinery, New York, NY, USA, 3--14. https://doi.org/10.1145/2486001.2486019
[20]
Pantea Karimi, Solal Pirelli, Siva Kesava Reddy Kakarla, Ryan Beckett, Santiago Segarra, Beibin Li, Pooria Namyar, and Behnaz Arzani. 2024. Towards Safer Heuristics With XPlain. arXiv preprint arXiv:2410.15086. https://arxiv.org/abs/2410.15086
[21]
Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. 2021. Reluplex: a Calculus for Reasoning about Deep Neural Networks. Formal Methods in System Design (2021).
[22]
Guy Katz, Derek A. Huang, Duligur Ibeling, Kyle D. Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zeljić, David L. Dill, Mykel J. Kochenderfer, and Clark W. Barrett. 2019. The Marabou Framework for Verification and Analysis of Deep Neural Networks. In International Conference on Computer Aided Verification.
[23]
Scott Kirkpatrick, D. Gelatt Jr., and Mario P. Vecchi. 1983. Optimization by Simulated Annealing. Science (1983).
[24]
Jitendra Kumar and Ashutosh Kumar Singh. 2019. Cloud Resource Demand Prediction using Differential Evolution based Learning. In 2019 7th International Conference on Smart Computing & Communications (ICSCC). 1--5. https://doi.org/10.1109/ICSCC.2019.8843680
[25]
Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. 2018. Adversarial examples in the physical world. In Artificial intelligence safety and security. Chapman and Hall/CRC, 99--112.
[26]
Ximeng Liu, Shizhen Zhao, and Yong Cui. 2024. FIGRET: Fine-Grained Robustness-Enhanced Traffic Engineering. ACM SIGCOMM (2024).
[27]
Alessio Lomuscio and Lalit Maganti. 2017. An approach to reachability analysis for feed-forward ReLU neural networks. (2017). http://arxiv.org/abs/1706.07351
[28]
Jiaqi Ma, Shuangrui Ding, and Qiaozhu Mei. 2020. Towards more practical adversarial attacks on graph neural networks. Advances in neural information processing systems 33 (2020), 4756--4766.
[29]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017).
[30]
Tanwi Mallick, Mariam Kiran, Bashir Mohammed, and Prasanna Balaprakash. 2020. Dynamic Graph Neural Network for Traffic Forecasting in Wide Area Networks. 2020 IEEE International Conference on Big Data (Big Data) (2020), 1--10.
[31]
Roland Meier, Thomas Holterbach, Stephan Keck, Matthias Stähli, Vincent Lenders, Ankit Singla, and Laurent Vanbever. 2019. (Self) Driving Under the Influence: Intoxicating Adversarial Network Inputs. In Proceedings of the 18th ACM Workshop on Hot Topics in Networks (Princeton, NJ, USA) (HotNets '19). 34--42.
[32]
Pooria Namyar, Behnaz Arzani, Ryan Beckett, Santiago Segarra, Himanshu Raj, and Srikanth Kandula. 2022. Minding the gap between fast heuristics and their optimal counterparts. In Proceedings of the 21st ACM Workshop on Hot Topics in Networks (HotNets '22). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3563766.3564102
[33]
Pooria Namyar, Behnaz Arzani, Ryan Beckett, Santiago Segarra, Himanshu Raj, Umesh Krishnaswamy, Ramesh Govindan, and Srikanth Kandula. 2024. Finding Adversarial Inputs for Heuristics using Multilevel Optimization. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, Santa Clara, CA, 927--949. https://www.usenix.org/conference/nsdi24/presentation/namyar-finding
[34]
Pooria Namyar, Behnaz Arzani, Srikanth Kandula, Santiago Segarra, Daniel Crankshaw, Umesh Krishnaswamy, Ramesh Govindan, and Himanshu Raj. 2024. Solving Max-Min Fair Resource Allocations Quickly on Large Graphs. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, Santa Clara, CA, 1937--1958. https://www.usenix.org/conference/nsdi24/presentation/namyar-solving
[35]
Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. 2019. Solving a class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems 32 (2019).
[36]
Yarin Perry, Felipe Vieira Frujeri, Chaim Hoch, Srikanth Kandula, Ishai Menache, Michael Schapira, and Aviv Tamar. 2023. DOTE: Rethinking (Predictive) WAN Traffic Engineering. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 1557--1581. https://www.usenix.org/conference/nsdi23/presentation/perry
[37]
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Semidefinite relaxations for certifying robustness to adversarial examples. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS'18).
[38]
Jason Ramapuram and Russ Webb. 2020. Improving Discrete Latent Representations With Differentiable Approximation Bridges. In 2020 International Joint Conference on Neural Networks (IJCNN). 1--10. https://doi.org/10.1109/IJCNN48605.2020.9207581
[39]
Eric Schulz, Maarten Speekenbrink, and Andreas Krause. 2018. A tutorial on Gaussian process regression: Modelling, exploring, and exploiting functions. Journal of mathematical psychology 85 (2018), 1--16.
[40]
Stanford University IT. 2015. Abilene Core Topology. https://uit.stanford.edu/service/network/internet2/abilene
[41]
Cheng Tan, Changliu Liu, Zhihao Jia, and Tianhao Wei. 2023. Building Verified Neural Networks for Computer Systems with Ouroboros. In Proceedings of Machine Learning and Systems.
[42]
Cheng Tan, Yibo Zhu, and Chuanxiong Guo. 2021. Building verified neural networks with specifications for systems. In Proceedings of the 12th ACM SIGOPS Asia-Pacific Workshop on Systems (Hong Kong, China) (APSys '21). 42--47.
[43]
Vincent Tjeng, Kai Xiao, and Russ Tedrake. [n. d.]. Evaluating Robustness of Neural Networks with Mixed Integer Programming. ICLR 2019 ([n.d.]).
[44]
Asaf Valadarsky, Michael Schapira, Dafna Shahaf, and Aviv Tamar. 2017. Learning to Route. In Proceedings of the 16th ACM Workshop on Hot Topics in Networks (HotNets '17). Association for Computing Machinery, New York, NY, USA.
[45]
Zhengxu Xia, Yajie Zhou, Francis Y. Yan, and Junchen Jiang. 2022. Genet: automatic curriculum generation for learning adaptation in networking. In Proceedings of the ACM SIGCOMM 2022 Conference (SIGCOMM '22).
[46]
Zhiying Xu, Francis Y. Yan, Rachee Singh, Justin T. Chiu, Alexander M. Rush, and Minlan Yu. 2023. Teal: Learning-Accelerated Optimization of WAN Traffic Engineering. In Proceedings of the ACM SIGCOMM 2023 Conference (ACM SIGCOMM '23). Association for Computing Machinery, New York, NY, USA, 378--393. https://doi.org/10.1145/3603269.3604857
[47]
Francis Y. Yan, Hudson Ayers, Chenzhi Zhu, Sadjad Fouladi, James Hong, Keyi Zhang, Philip Levis, and Keith Winstein. 2020. Learning in situ: a randomized experiment in video streaming. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). USENIX Association, Santa Clara, CA, 495--511. https://www.usenix.org/conference/nsdi20/presentation/yan
[48]
Jin Y. Yen. 1971. Finding the K Shortest Loopless Paths in a Network. Management Science 17, 11 (1971), 712--716.
Index Terms
- End-to-End Performance Analysis of Learning-enabled Systems
Recommendations
Engineering cyber‐physical systems through performance‐based modelling and analysis: A case study experience report
AbstractThe process of engineering cyber‐physical systems (CPS) is inevitably challenging because of the intrinsic problem of merging the specification of different ensembles that indicate hardware, software, and physical aspects of such systems. This ...
- Engineering cyber‐physical systems (CPS) is challenging due to the intrinsic problem of merging hardware, software, and physical aspects.
- Modelling and analysing the performance characteristics of CPS requires the coexistence of multiple models to get ...
GradeML: Towards Holistic Performance Analysis for Machine Learning Workflows
ICPE '21: Companion of the ACM/SPEC International Conference on Performance EngineeringToday, machine learning (ML) workloads are nearly ubiquitous. Over the past decade, much effort has been put into making ML model-training fast and efficient, e.g., by proposing new ML frameworks (such as TensorFlow, PyTorch), leveraging hardware ...
Integrating Power and Cooling into Parallel Performance Analysis
ICPPW '10: Proceedings of the 2010 39th International Conference on Parallel Processing WorkshopsWe present a new, integrated approach to parallel performance analysis that integrates traditional application-oriented performance data with measurements of the physical runtime environment. We have developed the needed infrastructure for combined ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2024
394 pages
ISBN:9798400712722
DOI:10.1145/3696348
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 November 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- NSF (National Science Foundation)
- Israel Science Foundation
- United States - Israel Binational Science Foundation
Conference
HOTNETS '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 110 of 460 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 131Total Downloads
- Downloads (Last 12 months)131
- Downloads (Last 6 weeks)72
Reflects downloads up to 13 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in