See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/224992331

The FlowSimulator framework for massively parallel CFD applications

Conference Paper · June 2010

Source: DLR

CITATIONS READS

37 506

2 authors:

Michael Meinel Gunnar Einarsson

German Aerospace Center (DLR) German Aerospace Center (DLR)
29 PUBLICATIONS   67 CITATIONS    23 PUBLICATIONS   179 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

FlowSimulator View project

DLR Research Software Engineering View project

All content following this page was uploaded by Michael Meinel on 20 May 2014.

The user has requested enhancement of the downloaded file.

Para 2010 – State of the Art in Scientific and Parallel Computing – extended abstract no. 44
University of Iceland, Reykjavik, June 6–9 2010
http://vefir.hi.is/para10/extab/para10-paper-44.pdf

The FlowSimulator framework for massively parallel CFD

applications

Michael Meinel∗1 and Gunnar Ólafur Einarsson†2

1 German Aerospace Center (DLR), Simulation and Software Technology
2 German Aerospace Center (DLR), Center for Computer Applications in Aerospace Science and Engineering

Abstract In this paper the FlowSimulator framework for multi- acting on the aircraft structure, and the deflection of the
disciplinary computational fluid dynamics simulations on high structure due to those forces. The changes in the shape
performance computer platforms is described. An overview of of the aircraft have an impact on the aerodynamic perfor-
the provided functionality is given and possible benefits for tool
mance, and this information is something that a design en-
developers as well as design engineers are presented.
gineer would like to have available. Consequently, those
Keywords CFD, HPC, multi-disciplinary framework, coupled two applications, CFD and computational structural me-
simulation chanics (CSM) have to be coupled “in-the-loop”, first, to
determine the steady aeroelastic equilibrium, and second,
to determine the response of the whole system during a
1 Introduction flight maneuver. However, in order to simulate complete
Over the past several years the field of computational fluid maneuvers, an additional flight mechanics module has to
dynamics (CFD) evolved from being a stand alone disci- be incorporated into the workflow as well [9]. New disci-
pline to being a part of a complex multi-disciplinary sim- plines are also being evolved, like computational aeroa-
ulation task. Design engineers long to integrate a growing coustics (CAA), which need very accurate aerodynamic
number of tools into their production workflows as the pre- solutions to use as starting points to determine acoustic
cise prediction of an aircraft’s flight performance requires sources [6].
the simulation of the whole flight envelope.
1.2 Incorporation of common tools
1.1 Evolution of CFD workflows The classical CFD workflow usually consists of four steps
When computational fluid dynamics (CFD) came on in the in order to get from the inital geometry description to a
early 1970’s, it was a great challenge on its own. This solution of the flow field, as depicted in Figure 1:
new combination of physics, numerical mathematics and
computer science was supported by the availability of high • geometry design;
performance computer platforms [3].
Although mono-discipline CFD research has reached a • mesh generation;
relatively mature state, and the use of CFD in the aircraft
• evaluating the actual flow solution and
industry is increasing, physical modelling and algorithm
development still remain as challenging research topics as • postprocessing to extract the required information.
ever. The reduction of the computational time needed by
CFD software is one of the driving goals of current re- Starting point for any simulation is the geometry which
search activities, especially when considered from a finan- is usually the result of a computer aided design (CAD)
cial point of view. Due to the increased acceptance of CFD process. These parametrical curves and surfaces are then
in industrial process chains, the desire to combine CFD passed to a mesh generation program which tesselates the
with other disciplines in order to improve the overall de- surfaces and creates a spacial discretization.
sign of a given product has awakened. Thus, increasing Before the actual CFD solver can work on this numeri-
effort is being put into coupled simulations of the different cal domain, a preprocessing step is required to determine
disciplines that affect aviation systems [5]. some extra data such as the size of time steps to simulate
As an example of the need for multi-disciplinary simu- or invariant derived data as wall normals or local volu-
lations is the interaction between the aerodynamic forces mina. Although some CFD codes integrate this prepro-
cessing step, it is not unusual to have it as an extra tool.
∗ Email: michael.meinel@dlr.de With the preprocessed data, the actual CFD solver is
† Email: gunnar.einarsson@dlr.de able to calculate a flow field solution for the problem given.
 Figure 1: Classical workflow of CFD applications.

However, the output has to be further postprocessed to ex-

tract the relevant data such as integral values and to format
them in a way that can be displayed by one of several avail-
able visualization tools.

1.3 New challenges in aerodynamic simulations

In the classical workflow, the most computationally inten-
sive part is the CFD solver itself. The impact of the mesh
generation, pre- and postprocessing are usually negligible.
Thus the common practice for steady flows is to have the
tools exchange data using file in- and output. As different
tools and different vendors often use their own data for-
mats, each step may require the inclusion of format con-
verters in the workflow.
With the development of more complex workflows such
as depicted in Figure 2 the dominant operation shifted
from one tool (the CFD solver) to a combination of tools
(CFD, CSM solvers and flight mechanics in this case). Us-
ing the classical file-and-converter-strategy for communi-
cation between the tools may lead to a significant perfor-
mance drop, as these operations now have to be done “in-
the-loop” and may thus become, especially for massively
parallel applications, a performance drag.
Another observable effect is closely related to the use of
recently available parallel network file systems for cluster
computers: When some thousands of processes try to ac- Figure 2: Sample workflow for coupled CFD simula-
tion.
cess the file system at the same time, they produce a large
amount of network traffic which slows down the overall
performance of the cluster and might even bring the com-
a software architecture as sketched in Figure 3 [4].
plete file system down. This depends, of course, a lot on
how much of the cluster hardware is dedicated to file sys- 2.1 Python as integration environment
tem handling instead of computational resources.
In order to provide easy extensibility of the FlowSimulator,
2 FlowSimulator software architecture a plugin system has been implemented. Instead of rein-
The FlowSimulator software, as initiated by Airbus, venting the wheel, Python was chosen as the integration
EADS1 Military Air Systems, and several European Re- environment [8]. Although this approach requires that the
search Institutes, is designed to be a platform where multi- respective tools might have to be slightly modified in order
disciplinary simulations can be run on massively parallel to allow a connection to Python, the assumption is that the
cluster architectures as efficiently as possible. The soft- benefits far outweigh the effort required for such modifica-
ware is intended to address the current and future needs tions.
of design engineers who require multi-disciplinary analy- Over the past several years, Python has gained a lot of
sis in their work. The FlowSimulator platform establishes acceptance within the scientific community, not only due
to great packages like NumPy and SciPy [2], but also for
1 The European Aearonautic Defence and Space Company (EADS) is the ease with which it can be used as the “glue” between
the biggest company in Europe in the fields of aeronautics, space and diverse software components. With these packages and
defence technology. qualities, Python allows for the rapid development of pro-
 Figure 3: Basic architecture of the FlowSimulator framework.

totypes which integrate almost ad-hoc into the FlowSimu- pendence can be easily exploited to create a workflow us-
lator environment. ing tools that previously had no common interface through
Thanks to tools like SWIG2 [10] and f2py3 , legacy which they could share data.
codes are easily converted to Python modules which can Yet another positive aspect of the integration with the
then be loaded at runtime, thus providing a dynamic plug- FSDM is that the integration can be done without link-
in infrastructure. ing the tool against a proprietary library, thus avoiding any
type of licensing costs. Moreover, since any given tool
2.2 An open framework instead of a middleware li- is treated as a plugin by the FSDM and the core of the
brary FlowSimulator does not depend on any of the plugins, the
In order to more easily reach a wide range of tool devel- question of availability and licensing for individual plugins
opers, an extensive open interface to the FlowSimulator is completely at the discretion of the plugin owner.4
DataManager (FSDM) is provided, instead of using a com-
mon library which would have to be integrated in each tool. 2.3 A massive parallel environment
The interface is realized as a collection of C++ classes Current high performance computers are mainly built as
that are all wrapped to Python, leaving the choice open distributed memory systems with some kind of cluster ar-
to the tool designer of how to best integrate their code with chitecture. Thus, modern high performance applications
the FSDM: through the Python layer of FSDM, or directly need to be parallelized in order to use the resources of
with the C++ classes. these achitectures as efficiently as possible. Although par-
The open interface approach of integrating a tool with titioned global address space (PGAS) systems are emerg-
the framework should lead to a reduction in the cost of the ing, the most common backend for data-parallel applica-
integration, since extensive modifications of the existing tions is still the Message Passing Interface (MPI) [7].
code are not required. In general, only a new import and In order to provide an efficient environment for paral-
export module that interfaces with the FSDM has to be lel applications, the FlowSimulator framework is designed
implemented. As a consequence, the exchange of data be- to operate natively on massively parallel cluster architec-
tween one tool and another can now take place completely tures. The FSDM, as the backbone of the FlowSimulator,
without file I/O, since all tools can read and write directly is implemented to take full advantage of a parallel envi-
to the FSDM. ronment. Every data structure defined in the base library
Another positive effect of the integration with the FSDM knows its own communication context and provides con-
is that the import and export filter of one tool require no in- venient methods to easily (re-)distribute the data in a non-
formation on the data format required by any of the other blocking manner.
tools being used in a given workflow. Since each tool im- This built-in parallelism of the FSDM supports the par-
ports and exports data directly to the FSDM, the data trans- allel tools in a given CFD workflow to access data without
fer between the tools is completely independent of the in- the need of a central coordination instance. The redistribu-
ternal data format used by the individual tools. This inde- tion of the data according to the needs of the tools can be
2 SWIG is a software development tool that connects programs written
easily achieved within the FSDM and is optimized for the
in C and C++ with a variety of high-level programming languages.
3 f2py is part of the NumPy distribution and allows automatic integra- 4 Although provided at no cost to project partners, the FSDM is not

tion of Fortran codes with Python. provided under an open source license.
 layout of the provided data structures. • visualization tools for on-line visualization of the
simulation results.
2.4 Included base functionality
FSDM comes with some basic tools already built in. Most Modules which are expected to be integrated in the near
important of these is a wide range of import and export fil- future include
ters that can handle most of the common file formats used
in CFD such as CGNS[1], NetCDF[12] and Tecplot[11]. • aeroacoustics codes and
As with all FSDM modules, the import and export fil- • special domain tools like buffeting simulations.
ters are fully parallel from the ground up, allowing large
datasets to be handled with ease, as there is no need to The FlowSimulator has already been established at in-
store all of the data in the memory of a single compute dustrial partners, and first production workflows on high
node at any time. performance computers have been implemented.
Another important functionality closely related to the
import and export filters are the built in partitioning al- References
gorithms. These allow the distribution of the data inde- [1] CFD General Notation System.
pendent from the tools that are to use them and allows a http://cgns.sourceforge.net.
domain decomposition that is invariant from the tool to be
[2] NumPy / SciPy homepage. http://scipy.org.
used.
Other tools, like some geometry operations (transforma- [3] J. Blazek. Computational Fluid Dynamics: Principles and
tion and basic tesselation), as well as a basic linear algebra Applications. Elsevier Ltd., second edition, 2005.
module are provided in addition. [4] H. Bleeke (Airbus) and U. Tremel (EADS MAS).
FlowSimulator Review Meeting. November 2006.
3 Conclusion [5] S. Cant. High-performance compuing in computational
The FlowSimulator provides an open framework that can fluid dynamics: progress and challenges. Philosophical
be used by tool developers for integrating their software Transactions of the Royal Society A Mathematical, Phys-
into a common environment that has been designed to en- ical & Engineering Sciences, 360:1211–1225, 2002.
hance CFD related, multi-disciplinary numerical simula- [6] R. Ewert. Die Simulation von Strömungslärm bei
tions on massively parallel high-performance computer ar- Verkehrsflügzeugen. In Fortschritte der Akustik, volume
chitectures. The plugin enabled framework allows design I, pages 21–22, 2006.
engineers to select the tools they wish to integrate into their [7] MPI-Forum. Message Passing Interface.
specific multi-disciplinary workflow chain, treating each http://www.mpi-forum.org/docs.
tool as a black box, since all data transfer and conversion
[8] Python. Python website. http://www.python.org.
is taken care of by the FlowSimulator.
This does not only allow for easy combination of tools [9] A. Schütte, G. Einarsson, A. Raichle, B. Schöning, M. Orlt,
into workflows but also provides the opportunity to easily J. Neumann, J. Arnold, W. Mönnich, and T. Forkert. Nu-
replace a tool or to have a fair comparison between differ- merical simulation of maneuvering aircraft by aerody-
namic, flight mechanics and structural mechanics coupling.
ent tools as the environment is consistent. In 45th AIAA Aerospace Sciences Meeting and Exhibit, vol-
As the import and export of files is managed by the ume AIAA 2007-1070, Reno, Nevada, January 2007.
FlowSimulator DataManager, a tool that is integrated into
[10] SWIG. Simplified Wrapper and Interface Generator home-
the framework can use virtually every file format already
page. http://swig.org.
integrated in the FSDM.
The list of tools already integrated in the FlowSimula- [11] Tecplot. Plotting and Data Visualization Software.
tor contains most of the tools used in current production http://www.tecplot.com
environments, and the list is constantly growing. Some ex- [12] Unidata. NetCDF.
amples of integrated tools are http://www.unidata.ucar.edu/software/netcdf.

• import and export filters for the most common geom-

etry and mesh formats;

• a mesh generator;

• structured and unstructured CFD solvers;

• a structural mechanics code;

• a flight mechanics module;

• different pre- and postprocessors and

View publication stats