Abstract
With the development of smart grid and big data, massive data retrieval has received intensive attentions by practitioners and developers, especially for its using in power information collection applications. It is a tricky task for improving the retrieval efficiency of massive data. The main big data solutions are likely to store the data in Hbase, which is difficult to improve the performance when querying small data from a large volume of data. Existing methods for this problem are inclined to use phoenix to query Hbase, but they exhibit inefficient under multiple table joining. In this paper, to sovle the quick retrieval for massive data in power information collection systems, we present a method via using Elasticsearch and Hbase to augment the performance of massive data retrieval. Specifically, we store the Rowkey and index information in elasticsearch to help accelerate the querying of Hbase. Then an experiment in real use case is used to verify the effectiveness of presented method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abe, R., Taoka, H., McQuilkin, D.: Digital grid: communicative electrical grids of the future. IEEE Trans. Smart Grid 2(2), 399–410 (2011). https://doi.org/10.1109/TSG.2011.2132744
Al-Badarneh, A., Najadat, H., Al-Soud, M., Mosaid, R.: Phoenix: a mapReduce implementation with new enhancements. In: 2016 7th International Conference on Computer Science and Information Technology (CSIT), pp. 1–5 (2016). https://doi.org/10.1109/CSIT.2016.7549451
Cao, C., Wang, W., Zhang, Y., Lu, J.: Embedding index maintenance in store routines to accelerate secondary index building in HBase. In: 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), pp. 500–507 (2018). https://doi.org/10.1109/CLOUD.2018.00070
Cao, Y., Wang, B., Zhao, W., Zhang, X., Wang, H.: Research on searching algorithms for unstructured grid remapping based on KD tree. In: 2020 IEEE 3rd International Conference on Computer and Communication Engineering Technology (CCET), pp. 29–33 (2020). https://doi.org/10.1109/CCET50901.2020.9213175
Guo, N., Su, Y., Yang, H.: Storage and indexing of big data for power distribution networks. In: 2018 5th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud)/2018 4th IEEE International Conference on Edge Computing and Scalable Cloud (EdgeCom), pp. 224–2243 (2018). https://doi.org/10.1109/CSCloud/EdgeCom.2018.00049
Li, L., Liu, W., Zhong, Z., Huang, C.: SP-phoenix: a massive spatial point data management system based on phoenix. In: 2018 IEEE 20th International Conference on High Performance Computing and Communications; IEEE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pp. 1634–1641 (2018). https://doi.org/10.1109/HPCC/SmartCity/DSS.2018.00266
Liu, H., Huang, F., Li, H., Liu, W., Wang, T.: A big data framework for electric power data quality assessment. In: 2017 14th Web Information Systems and Applications Conference (WISA), pp. 289–292 (2017). https://doi.org/10.1109/WISA.2017.29
Rongrong, S., Qing, L., Xin, S., Baifeng, N., Qiang, W.: Application of big data in power system reform. In: 2021 IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC), pp. 1340–1342 (2021). https://doi.org/10.1109/IPEC51340.2021.9421337
Touloupas, G., Konstantinou, I., Koziris, N.: Rasp: real-time network analytics with distributed nosql stream processing. In: 2017 IEEE International Conference on Big Data (Big Data), pp. 2414–2419 (2017). https://doi.org/10.1109/BigData.2017.8258198
Wu, H., et al.: A performance-improved and storage-efficient secondary index for big data processing. In: 2017 IEEE International Conference on Smart Cloud (SmartCloud), pp. 161–167 (2017). https://doi.org/10.1109/SmartCloud.2017.32
Zhao, F., Wang, G., Deng, C., Zhao, Y.: A real-time intelligent abnormity diagnosis platform in electric power system. In: 16th International Conference on Advanced Communication Technology, pp. 83–87 (2014). https://doi.org/10.1109/ICACT.2014.6778926
Zhu, Y., Xu, Q., Shi, H., Samsudin, J.: DS-index: a distributed search solution for federated cloud. In: 2016 IEEE International Conference on Networking, Architecture and Storage (NAS), pp. 1–4 (2016). https://doi.org/10.1109/NAS.2016.7549397
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, X., Hong, H., Wu, Z., Zeng, J., Liu, G. (2022). A Massive Data Retrieval Method for Power Information Collection Systems. In: Hu, B., Xia, Y., Zhang, Y., Zhang, LJ. (eds) Big Data – BigData 2022. BigData 2022. Lecture Notes in Computer Science, vol 13730. Springer, Cham. https://doi.org/10.1007/978-3-031-23501-6_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-23501-6_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23500-9
Online ISBN: 978-3-031-23501-6
eBook Packages: Computer ScienceComputer Science (R0)