research-article

SMARTMAP: operating system support for efficient data sharing among processes on a multi-core processor

Authors:

Ron Brightwell,

Kevin Pedretti,

Trammell HudsonAuthors Info & Claims

SC '08: Proceedings of the 2008 ACM/IEEE conference on Supercomputing

Article No.: 25, Pages 1 - 12

Published: 15 November 2008 Publication History

Abstract

This paper describes SMARTMAP, an operating system technique that implements fixed offset virtual memory addressing. SMARTMAP allows the application processes on a multi-core processor to directly access each other's memory without the overhead of kernel involvement. When used to implement MPI, SMARTMAP eliminates all extraneous memory-to-memory copies imposed by UNIX-based shared memory strategies. In addition, SMARTMAP can easily support operations that UNIX-based shared memory cannot, such as direct, in-place MPI reduction operations and one-sided get/put operations. We have implemented SMARTMAP in the Catamount lightweight kernel for the Cray XT and modified MPI and Cray SHMEM libraries to use it. Micro-benchmark performance results show that SMARTMAP allows for significant improvements in latency, bandwidth, and small message rate on a quad-core processor.

References

[1]

F. Cappello and D. Etiemble, "MPI versus MPI+OpenMP on IBM SP for the NAS benchmarks," in Proceedings of the ACM/IEEE International Conference on High-Performance Computing and Networking (SC '00), November 2000.

Digital Library

[2]

D. S. Henty, "Performance of hybrid message-passing and shared-memory parallelism for discrete element modeling," in Proceedings of the ACM/IEEE International Conference on High-Performance Computing and Networking (SC '00), November 2000.

Digital Library

[3]

S. Dong and G. E. Karniadakis, "Dual-level parallelism for deterministic and stochastic CFD problems," in Proceedings of the ACM/IEEE International Conference on High-Performance Computing and Networking (SC '02).

Digital Library

[4]

P. Druschel and L. L. Peterson, "Fbufs: A high-bandwidth cross-domain transfer facility," ACM SIGOPS Operating Systems Review, vol. 27, no. 5, pp. 189--202, December 1993.

Digital Library

[5]

D. Buntinas, G. Mercier, and W. Gropp, "Implementation and evaluation of shared-memory communication and synchronization operations in MPICH2 using the Nemesis communication subsystem," Parallel Computing, vol. 33, no. 9, pp. 634--644, September 2007.

Digital Library

[6]

D. Buntinas, G. Mercier, and W. Gropp, "Implementation and shared-memory evaluation of MPICH2 over the Nemesis communication subsystem," in Proceedings of the 2006 European PVM/MPI Users' Group Meeting, September 2006.

Digital Library

[7]

D. Buntinas, G. Mercier, and W. Gropp, "Design and evaluation of Nemesis, a scalable, low-latency, message-passing communication subsystem," in Proceedings of the 2006 International Symposium on Cluster Computing and the Grid, May 2006.

Digital Library

[8]

H.-W. Jin, S. Sur, L. Chai, and D. K. Panda, "Limic: Support for high-performance MPI intra-node communication on Linux," in Proceedings of the 2005 Cluster International Conference on Parallel Processing, June 2005.

Digital Library

[9]

K. Feind and K. McMahon, "An ultrahigh performance MPI implementation on SGI ccNUMA Altix systems," in Proceedings of the SGI Users' Group Technical Conference, June 2006.

[10]

D. Buntinas, G. Mercier, and W. Gropp, "Data transfers between processes in an smp system: Performance study and application to mpi," in Proceedings of the 2006 International Conference on Parallel Processing, August 2006.

Digital Library

[11]

L. Chai, P. Lai, H.-W. Jin, and D. K. Panda, "Designing an efficient kernel-level and user-level hybrid approach for MPI intranode communication on multi-core systems," in Proceedings of the International Conference on Parallel Processing, September 2008.

Digital Library

[12]

R. L. Graham and G. Shipman, "MPI support for multi-core architectures: Optimized shared memory collectives," in Proceedings of the 15th European PVM/MPI Users' Group Conference, September 2008.

Digital Library

[13]

R. Brightwell, T. Hudson, K. Pedretti, R. Riesen, and K. Underwood, "Implementation and performance of Portals 3.3 on the Cray XT3," in Proceedings of the 2005 IEEE International Conference on Cluster Computing, September 2005.

[14]

R. Brightwell, T. Hudson, K. T. Pedretti, and K. D. Underwood, "SeaStar interconnect: Balanced bandwidth for scalable performance," IEEE Micro, vol. 26, no. 3, May/June 2006.

Digital Library

[15]

S. M. Kelly and R. Brightwell, "Software architecture of the light weight kernel, Catamount," in Proceedings of the 2005 Cray User Group Annual Technical Conference, May 2005.

[16]

W. J. Camp and J. L. Tomkins, "Thor's hammer: The first version of the Red Storm MPP architecture," in In Proceedings of the SC 2002 Conference on High Performance Networking and Computing, Baltimore, MD, November 2002.

[17]

R. Brightwell, "A prototype implementation of MPI for SMARTMAP," in Proceedings of the 15th European PVM/MPI Users' Group Conference, September 2008.

Digital Library

[18]

Q. O. Snell, A. Mikler, and J. L. Gustafson, "NetPIPE: A network protocol independent performance evaluator," in Proceedings of the IASTED International Conference on Intelligent Information Management and Systems, June 1996.

Cited By

Gerofi BSantogidis AMartinet DIshikawa YZhao MChandra ARamakrishnan L(2018)PicoDriverProceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing10.1145/3208040.3208060(2-13)Online publication date: 11-Jun-2018
https://dl.acm.org/doi/10.1145/3208040.3208060
Hori ASi MGerofi BTakagi MDayal JBalaji PIshikawa YZhao MChandra ARamakrishnan L(2018)Process-in-processProceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing10.1145/3208040.3208045(131-143)Online publication date: 11-Jun-2018
https://dl.acm.org/doi/10.1145/3208040.3208045
Cho JJin HNam DPeña ABalaji PGropp WThakur R(2017)Enhanced memory management for scalable MPI intra-node communication on many-core processorProceedings of the 24th European MPI Users' Group Meeting10.1145/3127024.3127035(1-9)Online publication date: 25-Sep-2017
https://dl.acm.org/doi/10.1145/3127024.3127035
Show More Cited By

Index Terms

SMARTMAP: operating system support for efficient data sharing among processes on a multi-core processor

Recommendations

A Prototype Implementation of MPI for SMARTMAP
Proceedings of the 15th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface

Recently the Catamount lightweight kernel was extended to support direct access shared memory between processes running on the same compute node. This extension, called SMARTMAP, allows each process read/write access to another process' memory by ...
Optimizing Multi-core MPI Collectives with SMARTMAP
ICPPW '09: Proceedings of the 2009 International Conference on Parallel Processing Workshops

We recently enhanced a lightweight compute node operating system with a simple page table mapping strategy that allows parallel application processes within a node to share effectively a single address space. This strategy allows for each process to ...
OpenMP for Networks of SMPs

In this paper, we present the first system that implements OpenMP on a network of shared-memory multiprocessors. This system enables the programmer to rely on a single, standard, shared-memory API for parallelization within a multiprocessor and between ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '08: Proceedings of the 2008 ACM/IEEE conference on Supercomputing

November 2008

739 pages

ISBN:9781424428359

Conference Chair:
Patricia Teller

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

IEEE Press

Publication History

Published: 15 November 2008

Check for updates

Qualifiers

Research-article

Conference

SC '08

Sponsor:

SIGARCH
IEEE-CS

SC '08: International Conference for High Performance Computing, Networking, Storage and Analysis

November 15 - 21, 2008

Texas, Austin

Acceptance Rates

SC '08 Paper Acceptance Rate 59 of 277 submissions, 21%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
315
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gerofi BSantogidis AMartinet DIshikawa YZhao MChandra ARamakrishnan L(2018)PicoDriverProceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing10.1145/3208040.3208060(2-13)Online publication date: 11-Jun-2018
https://dl.acm.org/doi/10.1145/3208040.3208060
Hori ASi MGerofi BTakagi MDayal JBalaji PIshikawa YZhao MChandra ARamakrishnan L(2018)Process-in-processProceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing10.1145/3208040.3208045(131-143)Online publication date: 11-Jun-2018
https://dl.acm.org/doi/10.1145/3208040.3208045
Cho JJin HNam DPeña ABalaji PGropp WThakur R(2017)Enhanced memory management for scalable MPI intra-node communication on many-core processorProceedings of the 24th European MPI Users' Group Meeting10.1145/3127024.3127035(1-9)Online publication date: 25-Sep-2017
https://dl.acm.org/doi/10.1145/3127024.3127035
Evans NPedretti KKocoloski BLange JLang MBridges P(2016)A Cross-Enclave Composition Mechanism for Exascale System SoftwareProceedings of the 6th International Workshop on Runtime and Operating Systems for Supercomputers10.1145/2931088.2931094(1-8)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1145/2931088.2931094
Vaidyanathan KKalamkar DPamnany KHammond JBalaji PDas DPark JJoó BKern JVetter J(2015)Improving concurrency and asynchrony in multithreaded MPI applications using software offloadingProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/2807591.2807602(1-12)Online publication date: 15-Nov-2015
https://dl.acm.org/doi/10.1145/2807591.2807602
Kocoloski BLange JKielmann THildebrand DTaufer M(2015)XEMEMProceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing10.1145/2749246.2749274(89-100)Online publication date: 15-Jun-2015
https://dl.acm.org/doi/10.1145/2749246.2749274
Shimada AGerofi BHori AIshikawa YDaneshtalab MHemani APalesi M(2013)Proposing a new task model towards many-core architectureProceedings of the First International Workshop on Many-core Embedded Systems10.1145/2489068.2489075(45-48)Online publication date: 24-Jun-2013
https://dl.acm.org/doi/10.1145/2489068.2489075
Ma T(2012)Kernel-Assisted MPI Collective Communication among Many-core ClustersProceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)10.1109/CCGrid.2012.38(741-745)Online publication date: 13-May-2012
https://dl.acm.org/doi/10.1109/CCGrid.2012.38

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents