Abstract
This paper presents a new scheme, Send Gather Receive Scatter (SGRS), to perform zero-copy datatype communication over InfiniBand. This scheme leverages the gather/scatter feature provided by InfiniBand channel semantics. It takes advantage of the capability of processing non-contiguity on both send and receive sides in the Send Gather and Receive Scatter operations. In this paper, we describe the design, implementation and evaluation of this new scheme. Compared to the existing Multi-W zero-copy datatype scheme, the SGRS scheme can overcome the drawbacks of low network utilization and high startup costs. Our experimental results show significant improvement in both point-to-point and collective datatype communication. The latency of a vector datatype can be reduced by up to 62% and the bandwidth can be increased by up to 400%. The Alltoall collective benchmark shows a performance benefit of up to 23% reduction in latency.
This research is supported in part by Department of Energy’s Grant #DE-FC02-01ER25506, and National Science Foundation’s grants #CNS-0204429, and #CCR-0311542.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Ashworth, M.: A Report on Further Progress in the Development of Codes for the CS2. In: Carbonnell, F. (ed.) Deliverable D.4.1.b, EU DGIII, Brussels. GPMIMD2 ESPRIT Project (1996)
Byna, S., Sun, X.-H., Gropp, W., Thakur, R.: Improving the Performance of MPI Derived Datatypes by Optimizing Memory-Access Cost. In: Proceedings of the IEEE International Conference on Cluster Computing (2003)
Message Passing Interface Forum. MPI: A message-passing interface standard. The International Journal of Supercomputer Applications and High Performance Computing 8(3-4) (1994)
Fryxell, B., Olson, K., Ricker, P., Timmes, F.X., Zingale, M., Lamb, D.Q., Mac- Neice, P., Rosner, R., Tufo, H.: FLASH: An Adaptive Mesh Hydrodynamics Code for Modelling Astro physical Thermonuclear Flashes. Astrophysical Journal Suppliment 131, 273 (2000)
Gropp, W., Lusk, E., Swider, D.: Improving the Performance of MPI Derived Datatypes. In: MPIDC (1999)
InfiniBand Trade Association. InfiniBand Architecture Specification, Release 1.0, October 24 (2000)
Träff, J.L., Ritzdorf, H., Hempel, R.: The Implementation of MPI–2 One-sided Communication for the NEC SX. In: Proceedings of Supercomputing (2000)
Lawrence Berkeley National Laboratory. MVICH: MPI for Virtual Interface Architecture (August 2001)
Liu, J., Wu, J., Kini, S.P., Wyckoff, P., Panda, D.K.: High Performance RDMA-Based MPI Implementation over InfiniBand. In: 17th Annual ACM International Conference on Supercomputing (June 2003)
Lu, Q., Wu, J., Panda, D.K., Sadayappan, P.: Employing MPI Derived Datatypes to the NAS Benchmarks: A Case Study. Technical Report OSU-CISRC-02/04-TR10, Dept. of Computer and Information Science, The Ohio State University (February 2004)
Mellanox Technologies. Mellanox InifniBand Technologies, http://www.mellanox.com
Network-Based Computing Laboratory. MVAPICH: MPI for InfiniBand on VAPI Layer, http://nowlab.cis.ohio-state.edu/projects/mpi-iba/index.html
Ross, R., Miller, N., Gropp, W.: Implementing Fast and Reusable Datatype Processing. In: Dongarra, J., Laforenza, D., Orlando, S. (eds.) EuroPVM/MPI 2003. LNCS, vol. 2840, pp. 404–413. Springer, Heidelberg (2003)
Tipparaju, V., Santhanaraman, G., Nieplocha, J., Panda, D.K.: Host-Assisted Zero-Copy Remote Memory Access Communication on InfiniBand. In: IPDPS 2004 (April 2004)
Gropp, W., Lusk, E., Doss, N., Skjellum, A.: A High-Performance, Portable Implementation of the MPI, Message Passing Interface Standard
Wu, J., Wyckoff, P., Panda, D.K.: Supporting Efficient Noncontiguous Access in PVFS over InfiniBand. In: Proceedings of the IEEE International Conference on Cluster Computing (2003)
Wu, J., Wyckoff, P., Panda, D.K.: High Performance Implementation of MPI Datatype Communication over InfiniBand. In: International Parallel and Distributed Processing Symposium (IPDPS 2004) (April 2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Santhanaraman, G., Wu, J., Panda, D.K. (2004). Zero-Copy MPI Derived Datatype Communication over InfiniBand. In: Kranzlmüller, D., Kacsuk, P., Dongarra, J. (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface. EuroPVM/MPI 2004. Lecture Notes in Computer Science, vol 3241. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30218-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-540-30218-6_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23163-9
Online ISBN: 978-3-540-30218-6
eBook Packages: Springer Book Archive