Abstract
On Spark cloud computing platform, the conventional big data equi-join algorithms cannot meet the performance requirements well and the procedure of it is very time-consuming, so the efficiency of big data equi-join is a burning challenge. To overcome it, in this paper, we propose Compressed Bloom Filter Join algorithm, an efficient algorithm filters out most of invalid connections which cannot meet the criteria to reduce network overhead, and it constructs static one-dimensional bit array to improve join performance. Moreover, Compressed Bloom Filter Join Extension algorithm, an extended optimization based on Compressed Bloom Filter Join algorithm, produces a dynamic two-dimensional bit array to filter out invalid records, and it can further accelerate the process of data join when the data size is unknown. Experimental results show that the performance of two optimization algorithms which can reduce time consumption and the data size of Shuffle stage are better than Hash Join and Broadcast Join on Spark cloud computing platform.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Xin, R.: Spark and Scala (keynote). In: ACM SIGPLAN International Symposium on Scala, p. 1. ACM (2017)
Cui, Y., Li, G., Cheng, H., Wang, D.: Indexing for large scale data querying based on Spark SQL. In: IEEE International Conference on E-Business Engineering, pp. 103–108. IEEE (2017)
Zhang, J., Yang, Q., Shang, H., Zhang, H., Lin, Y., Zhou, R.: Performance evaluation for distributed join based on MapReduce. In: International Conference on Cloud Computing and Big Data, pp. 295–301. IEEE (2017)
Guo-Hua, L.I., Ren, Y.Q., Luo, C., Huang, J., Deng, Y.D.: Optimization of GPU-based main-memory hash join. In: IEEE International Conference on Computational Modeling, Simulation and Applied Mathematics (2017)
Sun, H.: Join processing and optimizing on large datasets based on hadoop framework (in Chinese). Dissertation, Nanjing University of Posts and Telecommunications (2013)
Lin, Y., Agrawal, D., Chen, C., Ooi, B.C., Wu, S.: Llama: leveraging columnar storage for scalable join processing in the MapReduce framework. In: ACM SIGMOD International Conference on Management of Data, pp. 961–972. ACM (2011)
Ramesh, S., Papapetrou, O., Siberski, W.: Optimizing distributed joins with bloom filters. In: Parashar, M., Aggarwal, S.K. (eds.) ICDCIT 2008. LNCS, vol. 5375, pp. 145–156. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89737-8_15
Zhang, C.C.: Design and optimize big-data join algorithms using MapReduce (in Chinese). Dissertation, University of Science and Technology of China (2014)
Huang, L.: Research on join query processing and optimization techniques in cloud computing environment (in Chinese). Dissertation, Liaoning University (2014)
Wei, L., Shen, Y., Su, C., Ooi, B.C.: Efficient processing of k nearest neighbor joins using MapReduce. Proc. VLDB Endow. 5(10), 1016–1027 (2012)
Blanas, S., Patel, J.M., Ercegovac, V., Rao, J., Shekita, E.J., Tian, Y.: A comparison of join algorithms for log processing in MaPreduce. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pp. 975–986. ACM (2010)
Zhang, L.: Research on query analysis and optimization based on spark system (in Chinese). Dissertation, Beijing Jiaotong University (2016)
Zhou, S.W.: Optimizing big data equi-join in spark and its application in analysis of network traffic data (in Chinese). Dissertation, South China University of Technology (2015)
Liu, R.C., Zhou, M.Q., Xing-Jie, P.I., Zhao, X.: Optimization of the equi-join problem based on big data in spark. Mod. Comput. 8, 3–6 (2017)
Zhong-Kui, H.U., Bo, Q.U., Huang, B., Wen-Yang, L.I.: A load balanced equi-join algorithm based on virtual processor range partition. Mod. Comput. (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, S., Xu, W. (2018). Big Data Equi-Join Optimization Algorithms on Spark Cloud Computing Platform. In: Sun, X., Pan, Z., Bertino, E. (eds) Cloud Computing and Security. ICCCS 2018. Lecture Notes in Computer Science(), vol 11063. Springer, Cham. https://doi.org/10.1007/978-3-030-00006-6_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-00006-6_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00005-9
Online ISBN: 978-3-030-00006-6
eBook Packages: Computer ScienceComputer Science (R0)