Runtime Variation in Big Data Analytics
Article No.: 67, Pages 1 - 20
Abstract
The dynamic nature of resource allocation and runtime conditions on Cloud can result in high variability in a job's runtime across multiple iterations, leading to a poor experience. Identifying the sources of such variation and being able to predict and adjust for them is crucial to cloud service providers to design reliable data processing pipelines, provision and allocate resources, adjust pricing services, meet SLOs and debug performance hazards.
In this paper, we analyze the runtime variation of millions of production Scope jobs on Cosmos, an exabyte-scale internal analytics platform at Microsoft. We propose an innovative 2-step approach to predict job runtime distribution by characterizing typical distribution shapes combined with a classification model with an average accuracy of >96%, using an innovative interpretable machine-learning algorithm out-performing traditional regression models and better capturing long tails. We examine factors such as job plan characteristics and inputs, resource allocation, physical cluster heterogeneity and utilization, and scheduling policies.
To the best of our knowledge, this is the first study on predicting categories of runtime distributions for enterprise analytics workloads at scale. Furthermore, we examine how our methods can be used to analyze what-if scenarios, focusing on the impact of resource allocation, scheduling, and physical cluster provisioning decisions on a job's runtime consistency and predictability.
Supplemental Material
MP4 File
Presentation video
- Download
- 211.65 MB
References
[1]
Amazon. 2022. Amazon EC2. https://aws.amazon.com/aws/ec2 Retrieved Feb 15, 2022 from
[2]
020)]% aws-athena, Amazon.com, Inc. 2020. Amazon Athena. https://aws.amazon.com/athena/ Retrieved July 4, 2020 from
[3]
Kanishka Bhaduri, Kamalika Das, and Bryan L Matthews. 2011. Detecting abnormal machine characteristics in cloud infrastructures. In 2011 IEEE 11th International Conference on Data Mining Workshops. IEEE, 137--144.
[4]
Christopher M Bishop and Nasser M Nasrabadi. 2006. Pattern recognition and machine learning. Vol. 4. Springer.
[5]
Eric Boutin, Paul Brett, Xiaoyu Chen, Jaliya Ekanayake, Tao Guan, Anna Korsun, Zhicheng Yin, Nan Zhang, and Jingren Zhou. 2015. Jetscope: Reliable and interactive analytics at cloud scale. Proceedings of the VLDB Endowment, Vol. 8, 12 (2015), 1680--1691.
[6]
Eric Boutin, Jaliya Ekanayake, Wei Lin, Bing Shi, Jingren Zhou, Zhengping Qian, Ming Wu, and Lidong Zhou. 2014. Apollo: Scalable and coordinated scheduling for cloud-scale computing. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14). 285--300.
[7]
Nicolas Bruno, Sameer Agarwal, Srikanth Kandula, Bing Shi, Ming-Chuan Wu, and Jingren Zhou. 2012. Recurring job optimization in scope. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. 805--806.
[8]
Nicolas Bruno, Sapna Jain, and Jingren Zhou. 2013. Continuous cloud-scale query optimization and processing. Proceedings of the VLDB Endowment, Vol. 6, 11 (2013), 961--972.
[9]
George Candea, Neoklis Polyzotis, and Radek Vingralek. 2011. Predictable performance and high query concurrency for data analytics. The VLDB Journal, Vol. 20, 2 (2011), 227--248.
[10]
Ronnie Chaiken, Bob Jenkins, Per-Åke Larson, Bill Ramsey, Darren Shakib, Simon Weaver, and Jingren Zhou. 2008. SCOPE: easy and efficient parallel processing of massive data sets. Proceedings of the VLDB Endowment, Vol. 1, 2 (2008), 1265--1276.
[11]
Roshan Chitrakar and Chuanhe Huang. 2012. Anomaly based intrusion detection using hybrid learning approach of combining k-medoids clustering and naive bayes classification. In 2012 8th International Conference on Wireless Communications, Networking and Mobile Computing. IEEE, 1--5.
[12]
Andrew Chung, Subru Krishnan, Konstantinos Karanasos, Carlo Curino, and Gregory R Ganger. 2020a. Unearthing inter-job dependencies for better cluster scheduling. In 14th $$USENIX$$ Symposium on Operating Systems Design and Implementation ($$OSDI$$ 20). 1205--1223.
[13]
Andrew Chung, Subru Krishnan, Konstantinos Karanasos, Carlo Curino, and Gregory R. Ganger. 2020b. Unearthing inter-job dependencies for better cluster scheduling. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 1205--1223.
[14]
Carlo Curino, Neha Godwal, Brian Kroth, Sergiy Kuryata, Greg Lapinski, Siqi Liu, Slava Oks, Olga Poppe, Adam Smiechowski, Ed Thayer, et al. 2020. MLOS: An infrastructure for automated software performance engineering. In Proceedings of the Fourth International Workshop on Data Management for End-to-End Machine Learning. 1--5.
[15]
Carlo Curino, Subru Krishnan, Konstantinos Karanasos, Sriram Rao, Giovanni M Fumarola, Botong Huang, Kishore Chaliparambil, Arun Suresh, Young Chen, Solom Heddaya, et al. 2019. Hydra: a federated resource manager for data-center scale analytics. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). 177--192.
[16]
Francesco Diaz and Roberto Freato. 2018. Azure Data Lake Store and Azure Data Lake Analytics. In Cloud Data Design, Orchestration, and Management Using Microsoft Azure. Springer, 327--392.
[17]
Songyun Duan, Shivnath Babu, and Kamesh Munagala. 2009. Fa: A system for automating failure diagnosis. In 2009 IEEE 25th International Conference on Data Engineering. IEEE, 1012--1023.
[18]
Zhiwei Fan, Rathijit Sen, Paraschos Koutris, and Aws Albarghouthi. 2020. Automated Tuning of Query Degree of Parallelism via Machine Learning. In Proceedings of the Third International Workshop on Exploiting Artificial Intelligence Techniques for Data Management. Article 2, 4 pages.
[19]
Dror G Feitelson and Bill Nitzberg. 1995. Job characteristics of a production parallel scientific workload on the NASA Ames iPSC/860. In workshop on job scheduling strategies for parallel processing. Springer, 337--360.
[20]
Andrew D Ferguson, Peter Bodik, Srikanth Kandula, Eric Boutin, and Rodrigo Fonseca. 2012. Jockey: guaranteed job latency in data parallel clusters. In Proceedings of the 7th ACM european conference on Computer Systems. 99--112.
[21]
Hans Fischer. 2011. A history of the central limit theorem: From classical to modern probability theory. Springer.
[22]
Eric Gaussier, David Glesser, Valentin Reis, and Denis Trystram. 2015. Improving backfilling by using machine learning to predict running times. In SC'15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1--10.
[23]
Ramanathan Gnanadesikan and Martin B Wilk. 1968. Probability plotting methods for the analysis of data. Biometrika, Vol. 55, 1 (1968), 1--17.
[24]
Gregg, Brendan. 2022. CPU Utilization is Wrong. https://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html Retrieved Oct 4, 2022 from
[25]
Jian Guo, Akihiro Nomura, Ryan Barton, Haoyu Zhang, and Satoshi Matsuoka. 2018. Machine learning predictions for underestimation of job runtime on HPC system. In Asian Conference on Supercomputing Frontiers. Springer, Cham, 179--198.
[26]
David Heckerman and John S Breese. 1996. Causal independence for probability assessment and inference using Bayesian networks. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, Vol. 26, 6 (1996), 826--831.
[27]
Herodotos Herodotou, Fei Dong, and Shivnath Babu. 2011. No One (Cluster) Size Fits All: Automatic Cluster Sizing for Data-Intensive Analytics (SOCC '11). Association for Computing Machinery, New York, NY, USA, Article 18, 14 pages. https://doi.org/10.1145/2038916.2038934
[28]
Zhiyao Hu, Dongsheng Li, Dongxiang Zhang, and Yixin Chen. 2020. ReLoca: Optimize Resource Allocation for Data-parallel Jobs using Deep Learning. In IEEE INFOCOM 2020 - IEEE Conference on Computer Communications. 1163--1171.
[29]
Jiamin Huang, Barzan Mozafari, Grant Schoenebeck, and Thomas Wenisch. 2016. Identifying the major sources of variance in transaction latencies: Towards more predictable databases. arXiv preprint arXiv:1602.01871 (2016).
[30]
Jiamin Huang, Barzan Mozafari, Grant Schoenebeck, and Thomas F Wenisch. 2017. A top-down approach to achieving performance predictability in database systems. In Proceedings of the 2017 ACM International Conference on Management of Data. 745--758.
[31]
Alekh Jindal, Hiren Patel, Abhishek Roy, Shi Qiao, Zhicheng Yin, Rathijit Sen, and Subru Krishnan. 2019. Peregrine: Workload Optimization for Cloud Query Engines. In Proceedings of the ACM Symposium on Cloud Computing. 416--427.
[32]
Sangeetha Abdu Jyothi, Carlo Curino, Ishai Menache, Shravan Matthur Narayanamurthy, Alexey Tumanov, Jonathan Yaniv, Ruslan Mavlyutov, Í nigo Goiri, Subru Krishnan, Janardhan Kulkarni, et al. 2016. Morpheus: Towards automated slos for enterprise clusters. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 117--134.
[33]
Konstantinos Karanasos, Sriram Rao, Carlo Curino, Chris Douglas, Kishore Chaliparambil, Giovanni Matteo Fumarola, Solom Heddaya, Raghu Ramakrishnan, and Sarvesh Sakalanaga. 2015. Mercury: Hybrid centralized and distributed scheduling in large shared clusters. In 2015 USENIX Annual Technical Conference (USENIX ATC 15). 485--497.
[34]
Marvin Karson. 1968. Handbook of Methods of Applied Statistics. Volume I: Techniques of Computation Descriptive Methods, and Statistical Inference. Volume II: Planning of Surveys and Experiments. IM Chakravarti, RG Laha, and J. Roy, New York, John Wiley; 1967, $9.00.
[35]
Michael Kuchnik, Jun Woo Park, Chuck Cranor, Elisabeth Moore, Nathan DeBardeleben, and George Amvrosiadis. 2019. This is why ML-driven cluster scheduling remains widely impractical. Tech. rep. (2019).
[36]
Po-Ling Loh and Peter Bühlmann. 2014. High-dimensional learning of linear causal networks via inverse covariance estimation. The Journal of Machine Learning Research, Vol. 15, 1 (2014), 3065--3105.
[37]
Gilles Louppe, Louis Wehenkel, Antonio Sutera, and Pierre Geurts. 2013. Understanding variable importances in forests of randomized trees. Advances in neural information processing systems, Vol. 26 (2013).
[38]
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems, Vol. 30 (2017).
[39]
Panagiotis Mandros, Mario Boley, and Jilles Vreeken. 2017. Discovering reliable approximate functional dependencies. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 355--363.
[40]
Microsoft Corporation. 2022. LGBMClassifier. https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html Retrieved Feb 4, 2022 from
[41]
Christoph Molnar. 2020. Interpretable machine learning. Lulu. com.
[42]
Mina Naghshnejad and Mukesh Singhal. 2018. Adaptive online runtime prediction to improve HPC applications latency in cloud. In 2018 IEEE 11th International Conference on Cloud Computing (CLOUD). IEEE, 762--769.
[43]
Frank Nielsen. 2016. Introduction to HPC with MPI for Data Science. Springer.
[44]
Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. 2019. Interpretml: A unified framework for machine learning interpretability. arXiv preprint arXiv:1909.09223 (2019).
[45]
Ilia Pietri, Gideon Juve, Ewa Deelman, and Rizos Sakellariou. 2014. A performance model to estimate execution time of scientific workflows on the cloud. In 2014 9th Workshop on Workflows in Support of Large-Scale Science. IEEE, 11--19.
[46]
Anish Pimpley, Shuo Li, Rathijit Sen, Soundararajan Srinivasan, and Alekh Jindal. 2022. Towards Optimal Resource Allocation for Big Data Analytics. In 25th International Conference on Extending Database Technology (EDBT). 338--350.
[47]
Anish Pimpley, Shuo Li, Anubha Srivastava, Vishal Rohra, Yi Zhu, Soundararajan Srinivasan, Alekh Jindal, Hiren Patel, Shi Qiao, and Rathijit Sen. 2021. Optimal Resource Allocation for Serverless Queries. arXiv preprint arXiv:2107.08594 (2021).
[48]
Conor Power, Hiren Patel, Alekh Jindal, Jyoti Leeka, Bob Jenkins, Michael Rys, Ed Triou, Dexin Zhu, Lucky Katahanas, Chakrapani Bhat Talapady, et al. 2021. The Cosmos Big Data platform at Microsoft: over a decade of progress and a decade to look forward. Proceedings of the VLDB Endowment, Vol. 14, 12 (2021), 3148--3161.
[49]
Shi Qiao, Adrian Nicoara, Jin Sun, Marc Friedman, Hiren Patel, and Jaliya Ekanayake. 2019. Hyper dimension shuffle: Efficient data repartition at petabyte scale in scope. Proceedings of the VLDB Endowment, Vol. 12, 10 (2019), 1113--1125.
[50]
Kaushik Rajan, Dharmesh Kakadia, Carlo Curino, and Subru Krishnan. 2016. PerfOrator: Eloquent Performance Models for Resource Optimization. In Proceedings of the Seventh ACM Symposium on Cloud Computing. 415--427.
[51]
Raghu Ramakrishnan, Baskar Sridharan, John R Douceur, Pavan Kasturi, Balaji Krishnamachari-Sampath, Karthick Krishnamoorthy, Peng Li, Mitica Manu, Spiro Michaylov, Rogério Ramos, et al. 2017. Azure data lake store: a hyperscale distributed file service for big data analytics. In Proceedings of the 2017 ACM International Conference on Management of Data. 51--63.
[52]
Garvesh Raskutti and Caroline Uhler. 2018. Learning directed acyclic graph models based on sparsest permutations. Stat, Vol. 7, 1 (2018), e183.
[53]
Lior Rokach and Oded Maimon. 2005. Clustering methods. In Data mining and knowledge discovery handbook. Springer, 321--352.
[54]
Aurobindo Sarkar and Amit Shah. 2018. Learning AWS: Design, build, and deploy responsive applications using AWS Cloud components. Packt Publishing Ltd.
[55]
Jörg Schad, Jens Dittrich, and Jorge-Arnulfo Quiané-Ruiz. 2010. Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance. PVLDB, Vol. 3, 1--2 (2010), 460--471. https://doi.org/10.14778/1920841.1920902
[56]
Scikit-Learn. 2022a. EnsembledClassifier. https://scikit-learn.org/stable/modules/ensemble.html Retrieved Feb 4, 2022 from
[57]
Scikit-Learn. 2022b. GaussianNB. https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html Retrieved Feb 4, 2022 from
[58]
Scikit-Learn. 2022c. GradientBoostingClassifier. https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html Retrieved Feb 4, 2022 from
[59]
Scikit-Learn. 2022d. GradientBoostingClassifier. https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html Retrieved Feb 4, 2022 from
[60]
Scikit-Learn. 2022 e. RandomForestClassifier. https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html Retrieved Feb 4, 2022 from
[61]
David Sculley. 2010. Web-scale k-means clustering. In Proceedings of the 19th international conference on World wide web. 1177--1178.
[62]
Rathijit Sen, Alekh Jindal, Hiren Patel, and Shi Qiao. 2020. AutoToken: Predicting Peak Parallelism for Big Data Analytics at Microsoft. PVLDB, Vol. 13, 12 (2020), 3326--3339.
[63]
Rathijit Sen, Abhishek Roy, Alekh Jindal, Rui Fang, Jeff Zheng, Xiaolei Liu, and Ruiping Li. 2021. AutoExecutor: predictive parallelism for spark SQL queries. Proceedings of the VLDB Endowment, Vol. 14, 12 (2021), 2855--2858.
[64]
Liqun Shao, Yiwen Zhu, Siqi Liu, Abhiram Eswaran, Kristin Lieber, Janhavi Mahajan, Minsoo Thigpen, Sudhir Darbha, Subru Krishnan, Soundar Srinivasan, et al. 2019. Griffon: Reasoning about Job Anomalies with Unlabeled Data in Cloud-based Platforms. In Proceedings of the ACM Symposium on Cloud Computing. 441--452.
[65]
S Shapley Ll. 1953. A value for n-person games. Contributions to the Theory of Games II, Annals of Mathematical Studies, Vol. 28 (1953).
[66]
Ozan Sonmez, Nezih Yigitbasi, Alexandru Iosup, and Dick Epema. 2009. Trace-based evaluation of job runtime and queue wait time predictions in grids. In Proceedings of the 18th ACM international symposium on High performance distributed computing. 111--120.
[67]
Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad Chakka, Suresh Anthony, Hao Liu, Pete Wyckoff, and Raghotham Murthy. 2009. Hive: a warehousing solution over a map-reduce framework. Proceedings of the VLDB Endowment, Vol. 2, 2 (2009), 1626--1629.
[68]
Jordan Tigani and Siddartha Naidu. 2014. Google BigQuery Analytics. John Wiley & Sons.
[69]
Dan Tsafrir, Yoav Etsion, and Dror G Feitelson. 2007. Backfilling using system-generated predictions rather than user runtime estimates. IEEE Transactions on Parallel and Distributed Systems, Vol. 18, 6 (2007), 789--803.
[70]
Philipp Unterbrunner, Georgios Giannikis, Gustavo Alonso, Dietmar Fauser, and Donald Kossmann. 2009. Predictable performance for unpredictable workloads. Proceedings of the VLDB Endowment, Vol. 2, 1 (2009), 706--717.
[71]
Dana Van Aken, Andrew Pavlo, Geoffrey J Gordon, and Bohan Zhang. 2017. Automatic database management system tuning through large-scale machine learning. In Proceedings of the 2017 ACM international conference on management of data. 1009--1024.
[72]
Vinod Kumar Vavilapalli, Arun C Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, Jason Lowe, Hitesh Shah, Siddharth Seth, et al. 2013. Apache hadoop yarn: Yet another resource negotiator. In Proceedings of the 4th annual Symposium on Cloud Computing. ACM, 5.
[73]
Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, Ion Stoica, et al. 2010. Spark: Cluster computing with working sets. HotCloud, Vol. 10, 10--10 (2010), 95.
[74]
Ji Zhang, Yu Liu, Ke Zhou, Guoliang Li, Zhili Xiao, Bin Cheng, Jiashu Xing, Yangtao Wang, Tianheng Cheng, Li Liu, et al. 2019. An end-to-end automatic cloud database tuning system using deep reinforcement learning. In Proceedings of the 2019 International Conference on Management of Data. 415--432.
[75]
Jiaxing Zhang, Hucheng Zhou, Rishan Chen, Xuepeng Fan, Zhenyu Guo, Haoxiang Lin, Jack Y Li, Wei Lin, Jingren Zhou, and Lidong Zhou. 2012. Optimizing data shuffling in data-parallel computation by understanding user-defined functions. In Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12). 295--308.
[76]
Yunjia Zhang, Zhihan Guo, and Theodoros Rekatsinas. 2020. A statistical perspective on discovering functional dependencies in noisy data. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 861--876.
[77]
Hai-Tao Zheng and Haiyang Zhang. 2015. Online feature selection based on passive-aggressive algorithm with retaining features. In Asia-Pacific Web Conference. Springer, 707--719.
[78]
Pengfei Zheng and Benjamin C Lee. 2018. Hound: Causal learning for datacenter-scale straggler diagnosis. Proceedings of the ACM on Measurement and Analysis of Computing Systems, Vol. 2, 1 (2018), 1--36.
[79]
Jingren Zhou, Nicolas Bruno, and Wei Lin. 2012. Advanced partitioning techniques for massively distributed computation. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. 13--24.
[80]
Jingren Zhou, Per-Ake Larson, and Ronnie Chaiken. 2010. Incorporating partitioning and parallel plans into the SCOPE optimizer. In 2010 IEEE 26th International Conference on Data Engineering (ICDE 2010). IEEE, 1060--1071.
[81]
Yiwen Zhu, Matteo Interlandi, Abhishek Roy, Krishnadhan Das, Hiren Patel, Malay Bag, Hitesh Sharma, and Alekh Jindal. 2021a. Phoebe: A Learning-Based Checkpoint Optimizer. Proc. VLDB Endow., Vol. 14, 11 (jul 2021), 2505--2518. https://doi.org/10.14778/3476249.3476298
[82]
Yiwen Zhu, Subru Krishnan, Konstantinos Karanasos, Isha Tarte, Conor Power, Abhishek Modi, Manoj Kumar, Deli Zhang, Kartheek Muthyala, Nick Jurgens, et al. 2021b. KEA: Tuning an Exabyte-Scale Data Infrastructure. In Proceedings of the 2021 International Conference on Management of Data. 2667--2680.
[83]
Dmitry Zotkin and Peter J Keleher. 1999. Job-length estimation and performance in backfilling schedulers. In Proceedings. The Eighth International Symposium on High Performance Distributed Computing (Cat. No. 99TH8469). IEEE, 236--243.
[84]
Salah Zrigui, Raphael Y de Camargo, Arnaud Legrand, and Denis Trystram. 2022. Improving the performance of batch schedulers using online job runtime classification. J. Parallel and Distrib. Comput., Vol. 164 (2022), 83--95.
Index Terms
- Runtime Variation in Big Data Analytics
Recommendations
Big data analytics in Cloud computing: an overview
AbstractBig Data and Cloud Computing as two mainstream technologies, are at the center of concern in the IT field. Every day a huge amount of data is produced from different sources. This data is so big in size that traditional processing tools are unable ...
Responsible Big Data Analytics for E-Business Services
ICBDR '21: Proceedings of the 5th International Conference on Big Data ResearchThis paper examines responsible big data analytics for e-business services and looks at how to use responsible big data analytics to obtain responsible e-business services. It addresses why responsibility matters to big data analytics and e-business ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2023
2807 pages
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 May 2023
Published in PACMMOD Volume 1, Issue 1
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 225Total Downloads
- Downloads (Last 12 months)106
- Downloads (Last 6 weeks)3
Reflects downloads up to 12 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in