Large-scale exploratory analysis, cleaning, and modeling for event detection in real-world power systems data
Article No.: 4, Pages 1 - 9
Abstract
In this paper, we present an approach to large-scale data analysis, Divide and Recombine (D&R), and describe a hardware and software implementation that supports this approach. We then illustrate the use of D&R on large-scale power systems sensor data to perform initial exploration, discover multiple data integrity issues, build and validate algorithms to filter bad data, and construct statistical event detection algorithms. This paper also reports on experiences using a non-traditional Hadoop distributed computing setup on top of a HPC computing cluster.
References
[1]
B. Akyol, J. Haack, B. Carpenter, S. Ciraci, M. Vlachopoulou, and C. Tews. Volttron: An agent execution platform for the electric power system. In Third International Workshop on Agent Technologies for Energy Systems Valencia, Spain, 2012.
[2]
D. B. Carr, R. J. Littlefield, W. Nicholson, and J. Littlefield. Scatterplot matrix techniques for large n. Journal of the American Statistical Association, 82(398): 424--436, 1987.
[3]
W. S. Cleveland and S. J. Devlin. Locally weighted regression: An approach to regression analysis by local fitting. Journal of the American Statistical Association, 83(403): 596--610, 1988.
[4]
Comprehensive R Archive Network. http://cran.r-project.org/, 2013. Accessed: 2013-04-10.
[5]
T. Dasu and T. Johnson. Exploratory data mining and data cleaning. Wiley-IEEE, 2003.
[6]
J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. Communications of the ACM, 51(1): 107--113, 2008.
[7]
S. Guha, R. Hafen, J. Rounds, J. Xia, J. Li, B. Xi, and W. S. Cleveland. Large complex data: divide and recombine (d&r) with rhipe. Stat, 1(1): 53--67, 2012.
[8]
Hadoop. http://hadoop.apache.org, 2013. Accessed: 2014-04-04.
[9]
R. Hafen. "datadr" github documentation. http://hafen.github.io/datadr/. Accessed: 2013-08-20.
[10]
R. Hafen, L. Gosink, J. McDermott, K. Rodland, K. Kleese-Van Dam, and W. S. Cleveland. Trelliscope: a system for detailed visualization in the deep analysis of large complex data. In Large Data Analysis and Visualization (LDAV), 2013 IEEE Symposium on. IEEE, 2013.
[11]
S. R. Jeffery, G. Alonso, M. J. Franklin, W. Hong, and J. Widom. A pipelined framework for online cleaning of sensor data streams. In Data Engineering, 2006. ICDE'06. Proceedings of the 22nd International Conference on, pages 140--140. IEEE, 2006.
[12]
A. Kleiner, A. Talwalkar, P. Sarkar, and M. I. Jordan. A scalable bootstrap for massive data. arXiv preprint arXiv:1112.5016, 2011.
[13]
G. M. Ljung and G. E. P. Box. On a measure of lack of fit in time series models. Biometrika, 65(2): 297--303, 1978.
[14]
E. Rahm and H. H. Do. Data cleaning: Problems and current approaches. IEEE Data Engineering Bulletin, 23(4): 3--13, 2000.
[15]
Revolution Analytics. http://www.revolutionanalytics.com, 2013. Accessed: 2013-04-10.
[16]
RHIPE. http://www.rhipe.org, 2013. Accessed: 2013-04-04.
[17]
Y. Zhang, P. Markham, T. Xia, L. Chen, Y. Ye, Z. Wu, Z. Yuan, L. Wang, J. Bank, J. Burgett, et al. Wide-area frequency monitoring network (fnet) architecture and applications. Smart Grid, IEEE Transactions on, 1(2): 159--167, 2010.
Index Terms
- Large-scale exploratory analysis, cleaning, and modeling for event detection in real-world power systems data
Recommendations
Big Data Analysis with Interactive Visualization using R packages
BigDataScience '14: Proceedings of the 2014 International Conference on Big Data Science and ComputingCompared to the traditional data storing, processing, analyzing and visualization which have been performed, Big data requires evolutionary technologies of massive data processing on distributed and parallel systems, such as Hadoop system. Big data ...
E-Learning Real Time Analysis Using Large Scale Infrastructure
BDCA'17: Proceedings of the 2nd international Conference on Big Data, Cloud and ApplicationsReal time data analytics is the ability to extract valuable information from live data. It represents a big opportunity to drive smart strategic decisions at the right time. The organizations which adopted this concept, such internet firms, have created ...
Big Data & Data Science: A Descriptive Research on Big Data Evolution and a Proposed Combined Platform by Integrating R and Python on Hadoop for Big Data Analytics and Visualization
ICCA 2020: Proceedings of the International Conference on Computing AdvancementsIn this technological era, Big Data is a new glorified term in where Data Science is the secret sauce of it. Undoubtedly, the digitalization of data is not the whole story; it is just a beginning of Data Science area of study. There was a time when the ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2013
49 pages
ISBN:9781450325103
DOI:10.1145/2536780
Copyright © 2013 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 17 November 2013
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Pacific Northwest National Laboratory Future Power Grid Initiative
Conference
SC13: International Conference for High Performance Computing, Networking, Storage and Analysis
November 17 - 22, 2013
Colorado, Denver
Acceptance Rates
HiPCNA-PG '13 Paper Acceptance Rate 5 of 7 submissions, 71%;
Overall Acceptance Rate 15 of 17 submissions, 88%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 168Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)1
Reflects downloads up to 09 Nov 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in