research-article

Free access

IoTAbench: an Internet of Things Analytics Benchmark

Authors:

Gowtham Bellala,

Ben VandiverAuthors Info & Claims

ICPE '15: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering

Pages 133 - 144

https://doi.org/10.1145/2668930.2688055

Published: 31 January 2015 Publication History

Abstract

The commoditization of sensors and communication networks is enabling vast quantities of data to be generated by and collected from cyber-physical systems. This ``Internet-of-Things" (IoT) makes possible new business opportunities, from usage-based insurance to proactive equipment maintenance. While many technology vendors now offer ``Big Data" solutions, a challenge for potential customers is understanding quantitatively how these solutions will work for IoT use cases. This paper describes a benchmark toolkit called IoTAbench for IoT Big Data scenarios. This toolset facilitates repeatable testing that can be easily extended to multiple IoT use cases, including a user's specific needs, interests or dataset. We demonstrate the benchmark via a smart metering use case involving an eight-node cluster running the HP Vertica analytics platform. The use case involves generating, loading, repairing and analyzing synthetic meter readings. The intent of IoTAbench is to provide the means to perform ``apples-to-apples" comparisons between different sensor data and analytics platforms. We illustrate the capabilities of IoTAbench via a large experimental study, where we store 22.8 trillion smart meter readings totaling 727 TB of data in our eight-node cluster.

References

[1]

A. Aji, F. Wang, H. Vo, R. Lee, Q. Liu, X. Zhang, and J. Saltz. Hadoop-GIS: A high performance spatial data warehousing system over MapReduce. In Proceedings of the VLDB Endowment. VLDB Endowment, August 2013.

Digital Library

[2]

AMTSybex and IBM. Sink or swim with smart meter data management. September 2011.

[3]

T. Armstrong, V. Ponnekanti, D. Borthakur, and M. Callaghan. LinkBench: a database benchmark based on the Facebook social graph. In SIGMOD 2013, pages 1185--1196. ACM, June 2013.

Digital Library

[4]

D. Borthakur. Petabyte scale databases and storage systems at Facebook. In SIGMOD 2013. ACM, June 2013.

Digital Library

[5]

B. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking cloud serving systems with YCSB. In SoCC'10, June 2010.

Digital Library

[6]

E. Dede, Madhusudhan, D. Gunter, R. Canon, and L. Ramakrishnan. Performance evaluation of a MongoDB and Hadoop platform for scientific data analysis. In ScienceCloud 2013, June 2013.

Digital Library

[7]

Z. Ding, X. Gao, J. Xu, and H. Wu. IOT-StatisticDB: A general statistical database cluster mechanism for big data analysis and the Internet of Things. In IEEE Internet of Things. IEEE, August 2013.

Digital Library

[8]

H. Dryar. The effect of weather on the system load. AIEE Transactions, 63(12):1006--1013, December 1944.

[9]

A. Floratou, N. Teletia, D. Dewitt, J. Patel, and D. Zhang. Can the elephants handle the NoSQL onslaught? In Proceedings of the VLDB Endowment, pages 1712--1723. VLDB Endowment, August 2012.

Digital Library

[10]

C. Gellings and R. Taylor. Electric load curve synthesis - a computer simulation of an electric utility load shape. IEEE Transactions on Power Apparatus and Systems, PAS-100(1):60--65, January 1981.

[11]

A. Ghazal, T. Rabl, M. Hu, F. Raab, M. Poess, A. Crolotte, and H. Jacobsen. BigBench: towards an industry standard benchmark for big data analytics. In SIGMOD 2013, pages 1197--1208. ACM, June 2013.

Digital Library

[12]

A. Hall, O. Bachmann, R. Bussow, S. Ganceanu, and M. Nunkesser. Processing a trillion cells per mouse click. In Proceedings of the VLDB Endowment, pages 1436--1446. VLDB Endowment, August 2012.

Digital Library

[13]

Itron and Microsoft. Benchmark testing results: Unparalleled scalability of Itron Enterprise Edition on SQL Server. May 2011.

[14]

D. Jones and M. Lorenz. An application of a Markov chain noise model to wind generator simulation. Mathematics and Computers in Simulation, 28:391--402, 1986.

Digital Library

[15]

S. Karnouskos, P. G. da Silva, and D. Illic. Assessment of high-performance smart metering for the web service enabled smart grid. In ICPE 2011, March 2011.

Digital Library

[16]

M. Kaufmann, A. Manjili, R. Vagenas, P. Fischer, D. Kossmann, F. Farber, and N. May. Timeline index: A unified data structure for processing queries on temporal data in SAP HANA. In SIGMOD 2013. ACM, June 2013.

Digital Library

[17]

A. Lamb, M. Fuller, R. Varadarajan, N. Tran, B. Vandiver, L. Doshi, and C. Bear. The Vertica analytic database: C-store 7 years later. In Proceedings of the VLDB Endowment, pages 1790--1801. VLDB Endowment, August 2012.

Digital Library

[18]

P. Larson, C. Clinciu, C. Fraser, E. Hanson, M. Mokhtar, R. Rusanu, and M. Saubhasik. Enhancements to SQL server column stores. In SIGMOD 2013. ACM, June 2013.

Digital Library

[19]

Y. Ma, J. Rao, W. Hu, X. Meng, X. Han, Y. Zhang, Y. Chai, and C. Liu. An efficient index for massive IOT data in cloud environment. In CIKM'12, October 2012.

Digital Library

[20]

C. MacGillvary, V. Turner, and D. Lund. Worldwide Internet of Things (IoT) 2013--2020 forecast: Billions of things, trillions of dollars. IDC, pages 1--22, October 2013.

[21]

Oracle. Meter-to-cash performance using Oracle Utilities applications on Oracle Exadata and Oracle Exalogic. January 2012.

[22]

S. Patil, M. Polte, K. Ren, W. Tantisiriroj, L. Xiao, J. Lopez, G. Gibson, A. Fuchs, and B. Rinaldi. YCSB

[23]

: benchmarking and performance debugging advanced features in scalable table stores. In SoCC'11, October 2011.

[24]

A. Pavlo, E. Paulson, A. Rasin, D. Abadi, D. DeWitt, S. Madden, and M. Stonebraker. A comparison of approaches to large-scale data analysis. In SIGMOD 2009. ACM, June 2009.

Digital Library

[25]

V. Raman, G. Attaluri, R. Barber, and N. Chainani. DB2 with BLU acceleration: so much more than just a column store. In Proceedings of the VLDB Endowment. VLDB Endowment, August 2013.

Digital Library

[26]

D. Shamshad, M. Bawadi, W. Hussin, T. Majid, and S. Samusi. First and second order Markov chain models for synthetic generation of wind speed time series. Energy, 30:693--708, 2005.

[27]

Y. Shi, X. Meng, J. Zhao, X. Hu, B. Liu, and H. Wang. Benchmarking cloud-based data management systems. In CloudDB'10, October 2010.

Digital Library

[28]

J. Shute, R. Vingralek, B. Samwel, B. Handy, C. Whipkey, E. Rollins, M. Oancea, K. Littlefield, D. Menestrina, S. Eliner, J. Cieslewicz, I. Rae, T. Stancescu, and H. Apte. F1: A distributed SQL database that scales. In Proceedings of the VLDB Endowment. VLDB Endowment, August 2013.

Digital Library

[29]

P. Wong, Z. He, and E. Lo. Parallel analytics as a service. In SIGMOD 2013, pages 25--36. ACM, June 2013.

Digital Library

[30]

R. Xin, J. Rosen, and M. Zaharia. Shark: SQL and rich analytics at scale. In SIGMOD 2013. ACM, June 2013.

Digital Library

Cited By

Khelifati AKhayati MDignös ADifallah DCudré-Mauroux P(2023)TSM-Bench: Benchmarking Time Series Database Systems for Monitoring ApplicationsProceedings of the VLDB Endowment10.14778/3611479.361153216:11(3363-3376)Online publication date: 24-Aug-2023
https://dl.acm.org/doi/10.14778/3611479.3611532
Saini AShende OPandit MSen RAnanthanarayanan GSha KBanerjee SChen J(2023)Bang for the Buck: Evaluating the cost-effectiveness of Heterogeneous Edge Platforms for Neural Network WorkloadsProceedings of the Eighth ACM/IEEE Symposium on Edge Computing10.1145/3583740.3628437(94-107)Online publication date: 6-Dec-2023
https://dl.acm.org/doi/10.1145/3583740.3628437
Passig Horstmann LWagner MFröhlich AVieira MCardellini VDi Marco ATuma P(2023)A Method to Evaluate the Performance of Predictors in Cyber-Physical SystemsProceedings of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578244.3583732(113-123)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578244.3583732
Show More Cited By

Index Terms

IoTAbench: an Internet of Things Analytics Benchmark

Recommendations

Analytics for the Internet of Things: A Survey

The Internet of Things (IoT) envisions a world-wide, interconnected network of smart physical entities. These physical entities generate a large amount of data in operation, and as the IoT gains momentum in terms of deployment, the combined scale of ...
BigBench: towards an industry standard benchmark for big data analytics
SIGMOD '13: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

There is a tremendous interest in big data by academia, industry and a large user base. Several commercial and open source providers unleashed a variety of products to support big data storage and processing. As these products mature, there is a need to ...
The role of big data analytics in Internet of Things

The explosive growth in the number of devices connected to the Internet of Things (IoT) and the exponential increase in data consumption only reflect how the growth of big data perfectly overlaps with that of IoT. The management of big data in a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '15: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering

January 2015

366 pages

ISBN:9781450332484

DOI:10.1145/2668930

General Chairs:
Lizy K. John
UT Austin, USA
,
Connie U. Smith
L&S Computer Technology, Inc., USA
,
Program Chairs:
Kai Sachs
SAP SE, Germany
,
Catalina M. Lladó
University of the Balearic Islands, Spain

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMETRICS: ACM Special Interest Group on Measurement and Evaluation
SIGSOFT: ACM Special Interest Group on Software Engineering
SPEC: SPEC Research Group

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 January 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICPE'15

Sponsor:

ICPE'15: ACM/SPEC International Conference on Performance Engineering

January 28 - February 4, 2015

Texas, Austin, USA

Acceptance Rates

ICPE '15 Paper Acceptance Rate 23 of 74 submissions, 31%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

49
Total Citations
View Citations
1,319
Total Downloads

Downloads (Last 12 months)162
Downloads (Last 6 weeks)24

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Khelifati AKhayati MDignös ADifallah DCudré-Mauroux P(2023)TSM-Bench: Benchmarking Time Series Database Systems for Monitoring ApplicationsProceedings of the VLDB Endowment10.14778/3611479.361153216:11(3363-3376)Online publication date: 24-Aug-2023
https://dl.acm.org/doi/10.14778/3611479.3611532
Saini AShende OPandit MSen RAnanthanarayanan GSha KBanerjee SChen J(2023)Bang for the Buck: Evaluating the cost-effectiveness of Heterogeneous Edge Platforms for Neural Network WorkloadsProceedings of the Eighth ACM/IEEE Symposium on Edge Computing10.1145/3583740.3628437(94-107)Online publication date: 6-Dec-2023
https://dl.acm.org/doi/10.1145/3583740.3628437
Passig Horstmann LWagner MFröhlich AVieira MCardellini VDi Marco ATuma P(2023)A Method to Evaluate the Performance of Predictors in Cyber-Physical SystemsProceedings of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578244.3583732(113-123)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578244.3583732
Zia MFrazier SNazaripouya H(2023)Synthetic Agricultural Load Data Generation Using TimeGANs2023 North American Power Symposium (NAPS)10.1109/NAPS58826.2023.10318596(1-6)Online publication date: 15-Oct-2023
https://doi.org/10.1109/NAPS58826.2023.10318596
Mondal SJayaraman PHassani AHaghighi PGeorgakopoulos D(2023)Situation-based Query Generation for Performance Evaluation of Cloud Managed IoT Applications2023 24th IEEE International Conference on Mobile Data Management (MDM)10.1109/MDM58254.2023.00064(352-357)Online publication date: Jul-2023
https://doi.org/10.1109/MDM58254.2023.00064
De Silva RZaslavsky ALoke SJayaraman P(2023)Context Query Generation using Scene Graph approach2023 24th IEEE International Conference on Mobile Data Management (MDM)10.1109/MDM58254.2023.00060(328-333)Online publication date: Jul-2023
https://doi.org/10.1109/MDM58254.2023.00060
Chio AJiang DGupta PBouloukakis GYus RMehrotra SVenkatasubramanian N(2023)SmartSPECPervasive and Mobile Computing10.1016/j.pmcj.2023.10180993:COnline publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1016/j.pmcj.2023.101809
Wang SBakir FEkaireb TPearson JKrintz CWolski R(2023)MSDBench: Understanding the Performance Impact of Isolation Domains on Microservice-Based IoT DeploymentsBenchmarking, Measuring, and Optimizing10.1007/978-3-031-31180-2_3(35-52)Online publication date: 13-May-2023
https://doi.org/10.1007/978-3-031-31180-2_3
KIM KSONG JKWAK J(2022)PRIGM: Partial-Regression-Integrated Generic Model for Synthetic Benchmarks Robust to Sensor CharacteristicsIEICE Transactions on Information and Systems10.1587/transinf.2021EDL8113E105.D:7(1330-1334)Online publication date: 1-Jul-2022
https://doi.org/10.1587/transinf.2021EDL8113
Korala HGeorgakopoulos DJayaraman PYavari A(2022)A Survey of Techniques for Fulfilling the Time-Bound Requirements of Time-Sensitive IoT ApplicationsACM Computing Surveys10.1145/351041154:11s(1-36)Online publication date: 9-Sep-2022
https://dl.acm.org/doi/10.1145/3510411
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents