Abstract
Aiming at the data skew problem in the Spark system caused by the unbalanced distribution of the input data and the default partition algorithm, this paper proposes an optimized partition method to solve the data skew problem. Firstly, the parallel cluster sampling algorithm is used to sample the intermediate data processed by each Map task to predict the data distribution. Then, the frequency of each Key is obtained according to the sampling prediction, and the weight is assigned to each Key. Finally, combining the greedy algorithm to divide the intermediate data to make the amount of data in each partition more balanced. Compared with the Hash and Range partitioning methods of the Spark platform and the SCID algorithm proposed by predecessors, experiments show this method effectively reduces the load deviation and reduces the task execution time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Hu, Y.H., Sheng, X., Mao, J.F.: Research on optimization algorithm for task scheduling in spark environment with unbalanced resources. Comput. Eng. Sci. 42(02), 203–209 (2020)
Liu, Z., Zhang, Q., Ahemd, R., et al.: Dynamic Resource Allocation for MapReduce with Partitioning Skew. IEEE Trans. Comput. 65(11), 3304–3317 (2016)
Xia, Y.C.: Research on Shuffle Mechanism of Spark Cluster. Chongqing University of Posts and Telecommunications (2017)
Zaman, S.K.U., Maqsood, T., Ali, M., et al.: A load balanced task scheduling heuristic for large-scale computing systems. Int. J. Comput. Syst. Sci. Eng. 34(02), 79–90 (2019)
Yan, Y.F.: Spark dynamic data partitioning algorithm based on Key-Value tilt model. Beijing University of Posts and Telecommunications (2019)
Zhang, Y.M., Jiang, J.B., Lu, J.W., et al.: Iterative data balancing partition strategy for MapReduce. Chinese J. Comput. 42(08), 1873–1885 (2019)
Jiang, J.B.: Research on MapReduce-oriented Intermediate Data Partitioning Strategy and Transmission Optimization. Zhejiang University of Technology (2019)
Zhang, Z.F., Wang, W.L., Geng, S.S., Jia, Z.T.: Research on spark data skew problem. J. Hebei Acad. Sci. 37(01), 1–7 (2020)
Tang, Z., Zhang, X.S., Li, K.L., Li, K.Q.: An intermediate data placement algorithm for load balancing in Spark computing environment. Future Gen. Comput. Systems 78(01), 287–301 (2016)
Wang, S.Z., Geng, S.S., Zhang, Z.F., et al.: A dynamic memory allocation optimization mechanism based on spark. Comput. Mat. Continua 58(02), 739–757 (2019)
Vengadeswaran, B.: Core–an optimal data placement strategy in hadoop for data intentitive applications based on cohesion relation. Int. J. Comput. Syst. Sci. Eng. 34(01), 47–60 (2019)
Huang, C.J.: Research on Data Balanced Distribution Algorithm in Spark. University of Electronic Science and Technology of China (2018)
Li, Q.Q.: Research on Spark task division and scheduling strategy for load balancing. Hunan University (2017)
Zhang, Li.: Research on Spark Load Balancing and Equivalent Join Optimization of Large Tables. Hebei University of Economics and Business (2019)
Xia, Z., Lu, L., Qin, T., et al.: A privacy-preserving image retrieval based on ac-coefficients and color histograms in cloud environment. Comput. Mat. Continua 58(01), 27–44 (2019)
Yang, Y., Zhao, Q., Ruan, L., et al.: Oversampling methods combined clustering and data cleaning for imbalanced network data. Intell. Autom. Soft Comput. 26(05), 1139–1155 (2020)
Acknowledgements
This paper is partially supported by the Social Science Foundation of Hebei Province (No. HB19JL007), and the Education technology Foundation of the Ministry of Education (No. 2017A01020).
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, S., Jia, Z., Wang, W. (2021). Research on Optimization of Data Balancing Partition Algorithm Based on Spark Platform. In: Sun, X., Zhang, X., Xia, Z., Bertino, E. (eds) Artificial Intelligence and Security. ICAIS 2021. Lecture Notes in Computer Science(), vol 12737. Springer, Cham. https://doi.org/10.1007/978-3-030-78612-0_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-78612-0_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78611-3
Online ISBN: 978-3-030-78612-0
eBook Packages: Computer ScienceComputer Science (R0)