In this paper, we introduce a collective communication abstraction layer which provides efficient collective communication operations on several common data ...
Harp: Collective Communication on Hadoop. Judy Qiu, Indiana University. Page 2. SA. Outline. • Machine Learning on Big Data. • Big Data Tools. • Iterative ...
By plugging Harp into Hadoop, we convert MapReduce model to Map-Collective model and enable efficient in-memory communication between map tasks across a variety ...
Harp: Collective Communication on Hadoop. Bingjing Zhang, Yang Ruan, Judy Qiu. Computer Science Department. Indiana University. Bloomington, IN, USA zhangbj ...
Harp is a HPC-ABDS (High Performance Computing Enhanced Apache Big Data Stack) framework aiming to provide distributed machine learning and other data intensive ...
In [33, 34] we introduce Harp as a plug-in to Hadoop with scientific data abstractions, support of iterations and high quality communication primitives. This ...
Bibliographic details on Harp: Collective Communication on Hadoop.
Harp is an implementation designed in a pluggable way to bring high performance to the Apache Big Data Stack and bridge the differences between Hadoop ecosystem ...
People also ask
Which Hive component is used to communicate with Hadoop framework?
What language is the Hadoop implementation Hadoop MapReduce written in?
The experiments show that with Harp we can scale three applications to 128 nodes with 4096 CPUs on the Big Red II supercomputer, where the speedup in most tests ...
This work considers the challenges of creating a high performance data analysis software framework in the context of the current HPC-ABDS software stack, ...