Abstract
In Big Data, SQL-on-Hadoop tools usually provide satisfactory performance for processing vast amounts of data, although new emerging tools may be an alternative. This paper evaluates if Apache Druid, an innovative column-oriented data store suited for online analytical processing workloads, is an alternative to some of the well-known SQL-on-Hadoop technologies and its potential in this role. In this evaluation, Druid, Hive and Presto are benchmarked with increasing data volumes. The results point Druid as a strong alternative, achieving better performance than Hive and Presto, and show the potential of integrating Hive and Druid, enhancing the potentialities of both tools.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
IBM, Zikopoulos, P., Eaton, C.: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data, 1st edn. McGraw-Hill Osborne Media (2011)
Ward, J.S., Barker, A.: Undefined by data: a survey of big data definitions. CoRR, abs/1309.5821 (2013)
Madden, S.: From databases to big data. IEEE Internet Comput. 16(3), 4–6 (2012)
Krishnan, K.: Data Warehousing in the Age of Big Data, 1st edn. Morgan Kaufmann Publishers Inc., San Francisco (2013)
Costa, C., Santos, M.Y.: Evaluating several design patterns and trends in big data warehousing systems. In: Krogstie, J., Reijers, H.A. (eds.) CAiSE 2018. LNCS, vol. 10816, pp. 459–473. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91563-0_28
Rodrigues, M., Santos, M.Y., Bernardino, J.: Big data processing tools: an experimental performance evaluation. Wiley Interdisc. Rev. Data Min. Knowl. Discov. 9, e1297 (2019)
Cuzzocrea, A., Bellatreche, L., Song, I.-Y.: Data warehousing and OLAP over big data: current challenges and future research directions. In: Proceedings of the Sixteenth International Workshop on Data Warehousing and OLAP, New York, USA, pp. 67–70 (2013)
Yang, F., Tschetter, E., Léauté, X., Ray, N., Merlino, G., Ganguli, D.: Druid: a real-time analytical data store. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 157–168 (2014)
Santos, M.Y., et al.: Evaluating SQL-on-Hadoop for big data warehousing on not-so-good hardware. In: ACM International Conference Proceeding Series, vol. Part F1294, pp. 242–252 (2017)
Costa, E., Costa, C., Santos, M.Y.: Partitioning and bucketing in hive-based big data warehouses. In: Rocha, Á., Adeli, H., Reis, L.P., Costanzo, S. (eds.) WorldCIST’18 2018. AISC, vol. 746, pp. 764–774. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77712-2_72
Chambi, S., Lemire, D., Godin, R., Boukhalfa, K., Allen, C.R., Yang, F.: Optimizing druid with roaring bitmaps. In: ACM International Conference Proceeding Series, 11–13 July 2016, pp. 77–86 (2016)
Correia, J., Santos, M.Y., Costa, C., Andrade, C.: Fast online analytical processing for big data warehousing. Presented at the IEEE 9th International Conference on Intelligent Systems (2018)
O’Neil, P.E., O’Neil, E.J., Chen, X.: The Star Schema Benchmark (SSB) (2009)
Kimball, R., Ross, M.: The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling. Wiley, Hoboken (2013)
LLAP - Apache Hive - Apache Software Foundation. https://cwiki.apache.org/confluence/display/Hive/LLAP. Accessed 07 Nov 2018
Druid Integration - Apache Hive - Apache Software Foundation. https://cwiki.apache.org/confluence/display/Hive/Druid+Integration. Accessed 07 Nov 2018
Ultra-fast OLAP Analytics with Apache Hive and Druid - Part 1 of 3, Hortonworks, 11 May 2017. https://hortonworks.com/blog/apache-hive-druid-part-1-3/. Accessed 07 Nov 2018
Acknowledgements
This work is supported by COMPETE: POCI-01-0145- FEDER-007043 and FCT – Fundação para a Ciência e Tecnologia within Project UID/CEC/00319/2013 and by European Structural and Investment Funds in the FEDER component, COMPETE 2020 (Funding Reference: POCI-01-0247-FEDER-002814).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Correia, J., Costa, C., Santos, M.Y. (2019). Challenging SQL-on-Hadoop Performance with Apache Druid. In: Abramowicz, W., Corchuelo, R. (eds) Business Information Systems. BIS 2019. Lecture Notes in Business Information Processing, vol 353. Springer, Cham. https://doi.org/10.1007/978-3-030-20485-3_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-20485-3_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20484-6
Online ISBN: 978-3-030-20485-3
eBook Packages: Computer ScienceComputer Science (R0)