research-article

Hybrid Transactional/Analytical Processing: A Survey

Authors:

Pinar TözünAuthors Info & Claims

SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data

Pages 1771 - 1775

https://doi.org/10.1145/3035918.3054784

Published: 09 May 2017 Publication History

Abstract

The popularity of large-scale real-time analytics applications (real-time inventory/pricing, recommendations from mobile apps, fraud detection, risk analysis, IoT, etc.) keeps rising. These applications require distributed data management systems that can handle fast concurrent transactions (OLTP) and analytics on the recent data. Some of them even need running analytical queries (OLAP) as part of transactions. Efficient processing of individual transactional and analytical requests, however, leads to different optimizations and architectural decisions while building a data management system.

For the kind of data processing that requires both analytics and transactions, Gartner recently coined the term Hybrid Transactional/Analytical Processing (HTAP). Many HTAP solutions are emerging both from the industry as well as academia that target these new applications. While some of these are single system solutions, others are a looser coupling of OLTP databases or NoSQL systems with analytical big data platforms, like Spark. The goal of this tutorial is to 1-) quickly review the historical progression of OLTP and OLAP systems, 2-) discuss the driving factors for HTAP, and finally 3-) provide a deep technical analysis of existing and emerging HTAP solutions, detailing their key architectural differences and trade-offs.

References

[1]

Apache Parquet. https://parquet.apache.org/.

[2]

R. Appuswarmy, M. Karpathiotakis, D. Porobic, and A. Ailamaki. The Case For Heterogeneous HTAP. In CIDR, 2017.

[3]

M. Armbrust, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. Bradley, X. Meng, T. Kaftan, M. J. Franklin, A. Ghodsi, and M. Zaharia. Spark SQL: Relational Data Processing in Spark. In SIGMOD, pages 1383--1394, 2015.

Digital Library

[4]

J. Arulraj, A. Pavlo, and P. Menon. Bridging the Archipelago Between Row-Stores and Column-Stores for Hybrid Workloads. In SIGMOD, pages 583--598, 2016.

Digital Library

[5]

R. Barber, C. Garcia-Arellano, R. Grosman, R. Mueller, V. Raman, R. Sidle, M. Spilchen, A. Storm, Y. Tian, P. Tözün, D. Zilio, M. Huras, G. Lohman, C. Mohan, F. Özcan, and H. Pirahesh. Evolving Databases for New-Gen Big Data Applications. In CIDR, 2017.

[6]

A. Boehm, J. Dittrich, N. Mukherjee, I. Pandis, and R. Sen. Operational analytics data management systems. PVLDB, 9:1601--1604, 2016.

Digital Library

[7]

P. Boncz, M. Zukowski, and N. Nes. MonetDB/X100: Hyper-Pipelining Query Execution. In CIDR, 2005.

[8]

Apache Cassandra. http://cassandra.apache.org.

[9]

A. Costea, A. Ionescu, B. Răaducanu, M. Switakowski, C. Bârca, J. Sompolski, A. Luszczak, M. Szafrański, G. de Nijs, and P. Boncz. Vectorh: Taking sql-on-hadoop to the next level. In SIGMOD '16, pages 1105--1117, 2016.

Digital Library

[10]

Danial Abadi and Shivnath Babu and Fatma Özcan and Ippokratis Pandis. Tutorial: SQL-on-Hadoop Systems. PVLDB, 8, 2015.

Digital Library

[11]

IBM dashDB. http://www.ibm.com/analytics/us/en/technology/cloud-data-services/dashdb.

[12]

DataStax Spark Cassandra Connector. https://github.com/datastax/spark-cassandra-connector.

[13]

C. Diaconu, C. Freedman, E. Ismert, P.-Å. Larson, P. Mittal, R. Stonecipher, N. Verma, and M. Zwilling. Hekaton: SQL Server's memory-optimized OLTP engine. In SIGMOD, pages 1243--1254, 2013.

Digital Library

[14]

F. Färber, N. May, W. Lehner, P. Große, I. Müller, H. Rauhe, and J. Dees. The SAP HANA Database -- An Architecture Overview. IEEE DEBull, 35(1):28--33, 2012.

[15]

S. Gray, F. Özcan, H. Pereyra, B. van der Linden, and A. Zubiri. IBM Big SQL 3.0: SQL-on-Hadoop without compromise. http://public.dhe.ibm.com/common/ssi/ecm/en/sww14019usen/SWW14019USEN.PDF, 2014.

[16]

SAP HANA Vora. http://go.sap.com/product/data-mgmt/hana-vora-hadoop.html.

[17]

Apache HBase. https://hbase.apache.org/.

[18]

Hive Transactions. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/hive-013-feature-transactions.html.

[19]

A. Kemper and T. Neumann. HyPer -- A Hybrid OLTP&OLAP Main Memory Database System Based on Virtual Memory Snapshots. In ICDE, pages 195--206, 2011.

Digital Library

[20]

M. Kornacker, A. Behm, V. Bittorf, T. Bobrovytsky, C. Ching, A. Choi, J. Erickson, M. Grund, D. Hecht, M. Jacobs, I. Joshi, L. Kuff, D. Kumar, A. Leblang, N. Li, I. Pandis, H. Robinson, D. Rorke, S. Rus, J. Russell, D. Tsirogiannis, S. Wanderman-Milne, and M. Yoder. Impala: A modern, open-source SQL engine for Hadoop. In CIDR, 2015.

[21]

Apache Kudu. https://kudu.apache.org/.

[22]

T. Lahiri, M.-A. Neimat, and S. Folkman. Oracle TimesTen: An In-Memory Database for Enterprise Applications. IEEE DEBull, 36(3):6--13, 2013.

[23]

A. Lamb, M. Fuller, R. Varadarajan, N. Tran, B. Vandiver, L. Doshi, and C. Bear. The Vertica Analytic Database: C-store 7 Years Later. PVLDB, 5(12):1790--1801, 2012.

Digital Library

[24]

MemSQL. http://www.memsql.com/.

[25]

C. Mohan. History Repeats Itself: Sensible and NonsenSQL Aspects of the NoSQL Hoopla. In EDBT, 2013.

Digital Library

[26]

B. Mozafari, J. Ramnarayan, S. Menon, Y. Mahajan, S. Chakraborty, H. Bhanawat, and K. Bachhav. SnappyData: A Unified Cluster for Streaming, Transactions and Interactice Analytics. In CIDR, 2017.

[27]

Apache ORC. https://orc.apache.org/.

[28]

A. Pavlo, J. Arulraj, L. Ma, P. Menon, T. C. Mowry, M. Perron, A. Tomasic, D. V. Aken, Z. Wang, and T. Zhang. Self-Driving Database Management Systems. In CIDR, 2017.

[29]

Apache Phoenix. http://phoenix.apache.org.

[30]

V. Raman, G. Attaluri, R. Barber, N. Chainani, D. Kalmuk, V. KulandaiSamy, J. Leenstra, S. Lightstone, S. Liu, G. M. Lohman, T. Malkemus, R. Mueller, I. Pandis, B. Schiefer, D. Sharpe, R. Sidle, A. Storm, and L. Zhang. DB2 with BLU Acceleration: So Much More than Just a Column Store. PVLDB, 6:1080--1091, 2013.

Digital Library

[31]

RocksDB. http://rocksdb.org/.

[32]

Roshan Sumbaly and others. Serving large-scale batch computed data with project Voldemort. In Proc. of the 10th USENIX conference on File and Storage Technologies, 2012.

Digital Library

[33]

Splice Machine. http://www.splicemachine.com/.

[34]

M. Stonebraker and U. Cetintemel. "One Size Fits All": An Idea Whose Time Has Come and Gone. In ICDE, pages 2--11, 2005.

Digital Library

[35]

M. Stonebraker and A. Weisberg. The VoltDB Main Memory DBMS. IEEE Data Eng. Bull., 36(2):21--27, 2013.

[36]

A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Anthony, H. Liu, and R. Murthy. Hive - A Petabyte Scale Data Warehouse Using Hadoop. In ICDE, 2010.

[37]

S. Tu, W. Zheng, E. Kohler, B. Liskov, and S. Madden. Speedy Transactions in Multicore In-memory Databases. In SOSP, pages 18--32, 2013.

Digital Library

[38]

Z. Zhang. Spark-on-HBase: Dataframe Based HBase Connector. http://hortonworks.com/blog/spark-hbase-dataframe-based-hbase-connector.

Cited By

Tamimi SBernhardt AStock FPetrov IKoch A(2024)DANSEN: Database Acceleration on Native Computational Storage by Exploiting NDPACM Transactions on Reconfigurable Technology and Systems10.1145/3655625Online publication date: 4-Apr-2024
https://doi.org/10.1145/3655625
Guo QAlizadeh MFalahati ABindschaedler LDing AAral AHilt V(2024)AlterEgoProceedings of the 7th International Workshop on Edge Systems, Analytics and Networking10.1145/3642968.3654814(7-12)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3642968.3654814
Zhang CLi GZhang JZhang XFeng J(2024)HTAP Databases: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.338969336:11(6410-6429)Online publication date: Nov-2024
https://doi.org/10.1109/TKDE.2024.3389693
Show More Cited By

Index Terms

Hybrid Transactional/Analytical Processing: A Survey
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Database query processing
      2. Database transaction processing

Recommendations

Adaptive HTAP through Elastic Resource Scheduling
SIGMOD '20: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

Modern Hybrid Transactional/Analytical Processing (HTAP) systems use an integrated data processing engine that performs analytics on fresh data, which are ingested from a transactional engine. HTAP systems typically consider data freshness at design ...
SnappyData: A Hybrid Transactional Analytical Store Built On Spark
SIGMOD '16: Proceedings of the 2016 International Conference on Management of Data

In recent years, our customers have expressed frustration in the traditional approach of using a combination of disparate products to handle their streaming, transactional and analytical needs. The common practice of stitching heterogeneous environments ...
HTAPBench: Hybrid Transactional and Analytical Processing Benchmark
ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

The increasing demand for real-time analytics requires the fusion of Transactional (OLTP) and Analytical (OLAP) systems, eschewing ETL processes and introducing a plethora of proposals for the so-called Hybrid Analytical and Transactional Processing (...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data

May 2017

1810 pages

ISBN:9781450341974

DOI:10.1145/3035918

General Chairs:
Rada Chirkova
North Carolina State University, USA
,
Jun Yang
Duke University, USA
,
Program Chair:
Dan Suciu
University of Washington, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 May 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS'17

Sponsor:

SIGMOD

SIGMOD/PODS'17: International Conference on Management of Data

May 14 - 19, 2017

Illinois, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
1,513
Total Downloads

Downloads (Last 12 months)82
Downloads (Last 6 weeks)4

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tamimi SBernhardt AStock FPetrov IKoch A(2024)DANSEN: Database Acceleration on Native Computational Storage by Exploiting NDPACM Transactions on Reconfigurable Technology and Systems10.1145/3655625Online publication date: 4-Apr-2024
https://doi.org/10.1145/3655625
Guo QAlizadeh MFalahati ABindschaedler LDing AAral AHilt V(2024)AlterEgoProceedings of the 7th International Workshop on Edge Systems, Analytics and Networking10.1145/3642968.3654814(7-12)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3642968.3654814
Zhang CLi GZhang JZhang XFeng J(2024)HTAP Databases: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.338969336:11(6410-6429)Online publication date: Nov-2024
https://doi.org/10.1109/TKDE.2024.3389693
Song HZhou WCui HPeng XLi F(2024)A survey on hybrid transactional and analytical processingThe VLDB Journal10.1007/s00778-024-00858-933:5(1485-1515)Online publication date: 4-Jun-2024
https://doi.org/10.1007/s00778-024-00858-9
Huynh AChaudhari HTerzi EAthanassoulis M(2024)Towards flexibility and robustness of LSM treesThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-023-00826-933:4(1105-1128)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s00778-023-00826-9
KUZNETSOV SVELIKHOV PFu Q(2023)REAL-TIME ANALYTICS: BENEFITS, LIMITATIONS, AND TRADEOFFSПрограммирование10.31857/S0132347423010053(3-31)Online publication date: 1-Jan-2023
https://doi.org/10.31857/S0132347423010053
Chen JShi RChen HZhang LLi RDing WFan LWang HXiong MChen YDong BGuo KLin YLiu XShi HWang PWang ZYang YZhao JZhou DZuo ZLiang Y(2023)Krypton: Real-Time Serving and Analytical SQL Engine at ByteDanceProceedings of the VLDB Endowment10.14778/3611540.361154516:12(3528-3542)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.14778/3611540.3611545
Song HZhou WLi FPeng XCui H(2023)Rethink Query Optimization in HTAP DatabasesProceedings of the ACM on Management of Data10.1145/36267501:4(1-27)Online publication date: 12-Dec-2023
https://dl.acm.org/doi/10.1145/3626750
Sarkar SPapon TStaratzis DZhu ZAthanassoulis M(2023)Enabling Timely and Persistent Deletion in LSM-EnginesACM Transactions on Database Systems10.1145/359972448:3(1-40)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.1145/3599724
Mancuso RRoozkhosh SHoornaert DMun JPapon TAthanassoulis M(2023)Software-Shaped PlatformsProceedings of Cyber-Physical Systems and Internet of Things Week 202310.1145/3576914.3587546(185-191)Online publication date: 9-May-2023
https://dl.acm.org/doi/10.1145/3576914.3587546
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents