Network and System Simulation Tools For Next Generation Networks: A Case Study
Network and System Simulation Tools For Next Generation Networks: A Case Study
Network and System Simulation Tools For Next Generation Networks: A Case Study
x5
1. Introduction
The modern information society will continue to emerge, and demand for wireless
communication services will grow. Future generation wireless networks are considered
necessary for the support of emerging services with their increasing requirements. Future
generation wireless networks are characterized by a distributed, dynamic, self-organizing
architecture (I. F. Akyildiz et al., 2006). These wireless networks are broadly categorized into
different wireless networks according to their specific characteristics. Typical examples
include Ad-Hoc/Mesh Networks, Sensor Networks, Cognitive Radio Networks, etc as
shown in figure 1. These wireless networks could then constitute the infrastructure of
numerous applications such as emergency and health-care systems, military, gaming,
advertisements, customer-to-customer applications, etc. Not only their importance in
military applications is growing, but also their impact on business is increasing. The
emergence of these wireless networks created many open issues in network design too.
More and more researchers are putting their efforts in designing the future generation
wireless networks.
1 Some part of this chapter was published at AMS’09, Indonesia (S.Mehta, et. al, 2009).
www.intechopen.com
82 Modelling, Simulation and Identiication
There are three main traditional techniques for analyzing the performance of wired and
wireless networks; analytical methods, computer simulation, and physical measurement or
a testbed measurement. Traditionally, formal modeling of systems has been via a
mathematical model, which attempts to find analytical solutions to problems and thereby
enable the prediction of the behavior of the system from a set of parameters and initial
conditions. However, it is widely known that comprehensive models for wireless ad hoc
networks are mathematically intractable. On its own, each individual layer of the protocol
stack may be complex enough to discourage attempts at mathematical analysis. Interactions
between layers in the protocol stack magnify this complexity. The construction of real
testbeds for any predefined scenario is usually an expensive or even impossible task, if
factors like mobility, testing area, etc. come into account. Additionally, most measurements
are not repeatable and require a high effort.
Simulation is, therefore, the most common approach to developing and testing new protocol
for a wireless network. Simulation has proven to be a valuable tool in many areas where
analytical methods aren’t applicable and experimentation isn’t feasible. Researchers
generally use simulation to analyze system performance prior to physical design or to
compare multiple alternatives over a wide range of conditions. In context with networks,
and especially wireless networks, simulators are used for the development and validation of
new algorithms, such as routing algorithms in wireless networks, or protocols.
Improvements of existing algorithms, as well as testing a networks capacity and efficiency
under specific scenarios is also a simulators task. Many publications typically include
performance simulations and commonly compare routing protocols. Simulators model the
real world in a specific way. Their purpose is to ease the understanding of it, to surge its
behavior and especially research its reactions on particular events. There are a number of
advantages to this approach: lower cost, ease of implementation, and practicality of testing
large-scale networks (J. Lessmann et. al, 2008).
The goal of simulators is to achieve an “as real as possible” situation in order to make the
simulation results realistic and therefore adaptable. Because it is impossible to collect and
implement all the data and details playing a role within the real world, the simulators have
to be trimmed. Now, the main difficulty is where to start cutting off details and where to
end with it while dealing with simulation. The correct level of detail decides whether a
simulation is useful or not, and therefore a difficult part in the development process. While
less details in simulation could produce results which are deluding or in some cases even
false, the effects of too many details can also make the simulation useless: Necessarily the
implementation is more time- consuming and the simulation takes longer. When it comes to
wireless network simulation, three main points are important: Firstly, the algorithms and
protocols should be error free and have to be implemented in adequate detail, and secondly,
the simulation environment, such as mobility schemes, must be realistic. Finally, a proper
method is needed to analyze the collected data. Even though simulation is a powerful tool, it
is still occupied with potential pitfalls (J. Heidemann et. al, 2001). To help overcome this, it is
important to know the different tools available and their benefit and drawbacks there in
associated. The goal of this paper is to give an over all short review to simulation system,
especially discussion about commonly used simulation tools in system and network, and a
cautionary guideline to avoid the pitfall associated with simulation for all who are using or
will be using simulation tools for their research.
www.intechopen.com
Network and System Simulation Tools for Next Generation Networks: A Case Study 83
There are four basic steps to run a simulation as shown in figure 2. First step is to develop a
model (e.g. implementation of a protocol); second step is to create a simulation scenario (e.g.
designing a network topology and traffic scenario); third step is to choose and collection of
statistics, and finally fourth step is to visualize and analysis of simulation results which may
be carried out after (or during, in some cases) the simulation execution. The problem with
such approach is that one cannot guess in advance how many replications is needed for
securing small errors of estimates, and if the errors are found to be too large, simulations
need to be repeated. This is referred to as offline sequential analysis of simulation output
data. Of course, this is not a very efficient way of data analysis. It is generally required that
final results from any simulation are to analyze output data on-line, during simulation.
Then, the simulation can be stopped when the statistical errors of the estimates become
sufficiently small.
Before going further, we present the classification of simulation tools which could be a good
summary for those with little previous exposure to the topic (S. M. Sanchez, 1999). As
depicted in figure 3 simulation tools can be classified according to several criteria including:
Stochastic or deterministic
Steady State or dynamic
Terminating or non terminating
Discrete or continuous or hybrid
Local or distributed
www.intechopen.com
84 Modelling, Simulation and Identiication
Stochastic simulation: Most of the realistic simulation tools The world is full of uncertainty,
and most (if not all) realistic simulation models will incorporate some randomness as well as
some element of time elapsing. Such tools can be used to examine a diverse set of
applications. For example, the simulation may have been designed to model the operation of
a customer service center, traffic patterns over a particular location grid, hospital facilities
utilization, waiting times for customers arriving at a service center, the number of cars
passing through an intersection during a 5 minute period, the efficacy of various strategies
in combat warfare, the impact of changes in layout and equipment on production
throughput, and more.
Deterministic simulation: Deterministic simulations use fixed, non-random values to
specify the model and particular variant of the system under investigation. Because there is
no randomness, the output is also fixed for any specific set of inputs. Chaotic model is the
special case of deterministic model.
Non-terminating simulation: In a non-terminating system, the duration of the system is not
finite. The Internet exemplifies a non-terminating system. Non-terminating simulations are
used to simulate non-terminating systems. In a non-terminating simulation, there is no
event to signal the end of a simulation, and such simulations are typically used to
investigate the long-term behavior of a system. Non-terminating simulations must, of
course, stop at some point, and it is a non-trivial problem to determine the proper duration
of a non-terminating simulation. If the behavior of the system becomes fairly stable at some
point, then there are techniques for analyzing the steady-state behavior of the system using
non-terminating simulations.
Terminating simulation: Terminating systems have fixed starting condition and a naturally
occurring event that marks the end of the system. An example of a terminating system is a
work day that starts at 9 am and ends at 6 pm at office. For terminating systems the initial
conditions of the system generally affect the desired measures of performance. The purpose
of simulating terminating systems is to understand transient behaviour of the system.
Steady-State simulation: Steady-state models use equations defining the relationships
between elements of the modeled system and attempt to find a state in which the system is
in equilibrium. Such models are often used in simulating physical systems, as a simpler
modeling case before dynamic simulation is attempted.
Dynamic simulation: Dynamic simulations model changes in a system in response to
(usually changing) input signals.
Discrite event simulation: A discrete event simulation manages events in time. Most
computer, logic-test and fault-tree simulations are of this type. In this type of simulation, the
simulator maintains a queue of events sorted by the simulated time they should occur. The
simulator reads the queue and triggers new events as each event is processed. It is not
important to execute the simulation in real time. It's often more important to be able to
access the data produced by the simulation, to discover logic defects in the design, or the
sequence of events. Most of the network simulation tools fall under this category.
Agent-Based Simulators: This is a special class of discrete event simulator in which the
mobile entities are known as agents. Whereas in a traditional discrete event model the
entities only have attributes, agents have both attributes and methods (e.g., rules for
interacting with other agents). An agent-based model could, for example, simulate the
behavior of a population of animals that are interacting with each other.
www.intechopen.com
Network and System Simulation Tools for Next Generation Networks: A Case Study 85
Continuous Simulators: These Simulators are opposite in nature to discrete simulators. This
class of tools solves differential equations that describe the evolution of a system using
continuous equations. These types of simulators are most appropriate for any information or
material which can be described as evolving or moving smoothly and continuously, rather
than in infrequent discrete steps or packets. For example, simulation of the movement of
water through a series of reservoirs and pipes can most appropriately be represented using
a continuous simulator.
Hybrid Simulators: These tools combine the features of continuous simulators and discrete
simulators. That is, they solve differential equations, but can superimpose discrete events
on the continuously varying system.
Distributed simulator: Distributed models run on a network of interconnected computers,
possibly through the Internet. Simulations dispersed across multiple host computers like
this are often referred to as distributed simulations.
Local simulator: Local simulator models run on an individual machine or within an
interconnected cluster.
2. Related Work
There are several surveys, comparisons, and also some case studies about wireless network
and system simulators. They all differ with respect to the selection of evaluated simulators,
the intention of the work, the focus of the potential comparison and the level of detail. Table
1 summarizes the previous related works.
Reference Type of Study Simulator Tools Scope of Study
(B. Schilling, 05’) Comparison Opnet, ns-2 Initialization, accuracy
Opnet,ns-2, QualNet, OMNeT++,
(S. Duflos et. al,
Comparison JSim, For critical infrastructure
06’)
SSFNet
ns-2, TOSSIM Models, visualization, architecture,
(M.Karl, 05’) Comparison
components
GloMoSim, ns-2, DIANEmu,
GTNetS,
(L. Hogie et al,
Description J-Sim, Jane, NAB, PDNS, Overview
05’)
OMNeT++,
Opnet, QualNet, SWANS
SSF, SWANS, J-Sim, NCTUns,
ns-2,
(E.E.lopez et al,
Comparison OMNeT++, Ptolemy, ATEMU, Models, type of visualization
06’)
Em-
Star, SNAP, TOSSIM
ns-2, GloMoSim, Opnet,
SensorSim,
J-Sim, Sense, OMNeT++, Sidh,
(D.Curren, 07’) Description Overview
Sens,
TOSSIM, ATEMU, Avrora,
EmStar
(L.Begg et al, Opnet, ns-2, OMNeT++, SSFNet, availability/credibility of models,
Comparison
06’) QualNet, J-Sim, Totem usability
(G.F. Lucio et al,
Case Study Opnet, ns-2, testbed Accuracy of results
03’)
(K. Pawlikowsk,
Survey In general simulation study Credibility , accuracy
02’)
Table 1. Related works on Simulator comparison
www.intechopen.com
86 Modelling, Simulation and Identiication
All of the works listed in table 1 consider different simulators or differ in their aim from this
paper. The works parented in (S. Duflos et. al, 06’, L. Hogie et al, 05’, E.E.lopez et al, 06’,
D.Curren, 07’, L.Begg et al, 06’, K. Pawlikowsk, 02’) are the close to our work as they include
some common simulators J-Sim, OMNeT++, and ns-2, which we also consider for our study.
However, (S. Duflos et. al, 2006’) examines their suitability for simulating the failure of
critical infrastructures like electricity or telecommunication networks. This is very unrelated
to what we present here. A huge list of simulators is presented in (L. Hogie et al, 2005,
D.Curren, 2007)however, they do not give a comparative study. Rather, their works consists
of more or less description of each simulator tools independently. In (E.E.lopez et al,
2006)authors give an overview about the different issues in wireless networks on a general
basis. Only at the end of their work they presented a table comparing the considered
simulation tools according to different features such as their language, the available
modules, and GUI support, etc. the most detailed comparison is presented in (L.Begg et al,
2006). However, they consider all the simulators from an industrial research point of view,
which are less relevant for academic researchers. They also miss several practical issues
regarding the credibility and reliability of the tools. In (K. Pawlikowsk, 2002) authors
presented a survey study of more then 2200 research papers in the field of network
simulation studies and point out several systematic flaws in that. We follow the similar kind
of work line of (K. Pawlikowsk, 2002) but with different aims. Our goal in this paper is to
make a basic contribution to the wireless network community by a) Giving overall short
overview of some widely used system and network simulation tools, b) comparing
simulation tools on the basis of several features and a survey report of more then 800
research papers in the field of system and networks in recent years (2000~2008), c) listing
our recommendations for the designers of protocols, models, and simulators.
The remainder of this paper is organized as follows. In Section 3, we provide a brief
overview on “widely used” network and system simulation tools, and their comparisons
and results from our survey. Finally, conclusions are presented in section 4.
2In rest of the paper we keep using terms “Physical Layer and “System” interchangeably,
unless and otherwise specified.
www.intechopen.com
Network and System Simulation Tools for Next Generation Networks: A Case Study 87
www.intechopen.com
88 Modelling, Simulation and Identiication
OMNet++ : OMNet++ (Object Modular network Testbed in C++) is well designed discrete
event simulation environment written in C++. OMNET++ is actually a general-purpose
simulator capable of simulating any system composed of devices interacting with each
others. The mobility extension for OMNeT++ is intended to support wireless and mobile
simulations within OMNeT++. This support is said to be fairly incomplete. OMNet++ is for
academic and educational use. Modules are connected in a hierarchical nested fashion,
where each module can contain several other modules. Modules can be defined as being
either simple or compound. Simple modules are used to define algorithms, and make up the
bottom of the hierarchy. Compound modules are a collection of simple modules that
interact with one another, using messages. OMNeT++ provides a component-based,
hierarchical, modular and extensible architecture. Components, or modules, are
programmed in C++ and new ones are developed using the C++ class library which consists
of the simulation kernel and utility classes for random number generation, statistics
collection, topology discovery etc. OMNeT++ has a number of advantages over the other
simulators. OMNeT++ accurately models most hardware and includes the modeling of
physical phenomena. All layers of the protocol stack can be modified. Despite its apparent
advantages, OMNeT++ has remained relatively obscure. The original implementation does
not offer a great variety of protocols, and very few have been implemented, leaving users
with significant background work if they want to test their own protocol in different
environments. OMNet++ works on Linux, Unix-like systems and windows XP/2K (OMnet,
web link).
OPNet: OPNet (Optimized Network Engineering Tools) Modeler is a discrete-event
network simulator first proposed by MIT in 1986 and is written in C++. It is a well
established and professional commercial suite for network simulation. It is actually the most
widely used commercial simulation environment. However, it can be used free of charge by
researchers applying to University Program of the product. Unlike ns-2 and GloMoSim,
OPNET supports the use of modeling different network-specific hardware, such as physical-
link transceivers and antennas. OPNET Modeler features an interactive development
environment allowing the design and study of networks, devices, protocols, and
applications. For this, an extensive list of protocols is supported. Particularly, MAC
protocols include IEEE 802.11a/b/g and Bluetooth ones.OPNET can also be used to define
custom packet formats. The simulator aids users in developing the various models through
a graphical interface. The interface can also be used to model, graph, and animate the
resulting output. One of the most interesting features of OPNet is its ability to execute and
monitor several scenarios in a concurrent manner. However, OPNET also suffers from the
same object-oriented scalability problems as ns-2. OPNet modeler runs on Windows XP/2K,
Linux and Solaris platforms (Opnet, web link).
QualNet: QualNet network simulation software has been developed and marketed by
Scalable Network technologies.. It is a commercial ad hoc network simulator based on the
GloMoSim. It provides a comprehensive set of tools with many components for custom
network modeling and simulation. Models in source code form provide developers with a
solid foundation from which to build new functionality or to modify exiting functionalities.
QualNet does have a range of wired as well as wireless models but its main strength is in
the wireless area. QualNet also largely extends the set of models and protocols supported
by the initial GloMoSim distribution. As it is built on top of GloMoSim, QualNet is written
in Parsec (Qualnet, web link).
www.intechopen.com
Network and System Simulation Tools for Next Generation Networks: A Case Study 89
3.1 Comparison
In this sub section we summarize the most interesting capabilities, advantages, and
drawbacks of existing tools for wireless networks in table 2. Table 2 has all simulators
considered in the previous section listed in the consecutive columns and special
features/capabilities in the context of all simulators in the consecutive rows, respectively.
Tools
GloMo-
Sr.N. Features NS2 J-Sim OMNet++ OPNet QualNet MATLAB
Sim
Applicabil Net./Sys Net./S Net./Sys System
1 Network Net./Sys. Net./Sys.
ity . ys. .
www.intechopen.com
90 Modelling, Simulation and Identiication
Parsec
C++/OT Parsec
2 Interface (C- Java/Jacl C++/NED C or C++ C++
cl (C-Based)
Based)
Data
Acquisition
Toolbox,
Instrument
Control
Toolbox ,
T/W/A T/W/Ad T/W/A T/W/Ad
Available T/W/ Image
3 d/ / T/W/Ad d/ /
Modules Ad Acquisition
WSNA WSNA WSN WSNA
Toolbox
Suppor Support
4 Mobility Support Support No Support Support
t
Excellent Excellent
Good Good Good
No or graphical graphical
Limited visualizatio visualization graphical
Graphical very support, support,
5 Visual n and and excellent support,
Support limited Excellent Excellent
aid debug facility for Excellent
visual aid facility for facility for
facility debug for debug.
debug. debug.
SMP
Parallelis RMI- SMP
6 No /Beow MPI/PVM Yes Yes
m based /Beowulf
ulf
Free
Free for
academic
Open Open Open academic and Commercia Commerci
7 License License for
Source Source Source educational l al
limited
use
use
Scalability Very
8 Small Large Small Large Medium Very Large
* Large
Document
ation and
9 Excellent Poor Poor Good Excellent Good Excellent
user
support
Extendibil Excelle
10 Excellent Excellent Excellent Excellent Excellent Excellent
ity* nt
Not Not
11 Emulation Limited Yes Limited Yes Yes
Direct Direct
Wireless Networks simulators exhibit different features and models. Each has advantages
and disadvantages, and each is appropriate in different situations. In choosing a simulator
from the available tools, the choice of a simulator should be driven by the requirements.
Developers must consider the pros and cons of different programming languages, the
means in which simulation is driven (event vs. time based), component-based or
objectoriented architecture, the level of complexity of the simulator, features to include and
not include, use of parallel execution, ability to interact with real nodes, and other design
choices. While design language choices are outside of the scope of this paper, there are some
guidelines that appear upon looking at a number of already existing simulators. Most
www.intechopen.com
Network and System Simulation Tools for Next Generation Networks: A Case Study 91
simulators use a discrete event engine for efficiency. Component-based architectures scale
significantly better than object-oriented architectures, but may be more difficult to
implement in a modularized way.
Defining each wired/wireless node as its own object ensures independence amongst the
nodes. The ease of swapping in new algorithms for different protocols also appears to be
easier in object-oriented designs. However, with careful programming, component based
architectures perform better and are more effective. Generally, the level of complexity built
into the simulator has a lot to do with the goals of the developers and the time constraints
imposed. Using a simple MAC protocol may suffice in most instances, and only providing
one saves significant amounts of time. If high-precision PHY layers are needed, then ns-2
(coupled with the highly-accurate PHY) is clearly the wisest choice. The number of nodes
targeted also determines the choice of the simulation tool. Sequential simulators should not
be expected to run more that 1,000 nodes. If larger scales are needed, then parallel
simulators are a wise choice. Finally, most non-commercial simulators suffer from a lack of
good documentation (NS2 is an exceptional case here) and support. Using a commercial one
might help in case of troubles. Moreover, commercial simulators usually feature extensive
lists of supported protocols, while open source solutions give full empowerment.
3.2 Analysis
In the previous section we provide the background on a number of different network simulators
and present the comparison of some important features of each. In continuation of our research
work, we present our survey results to take up on the credibility issues of simulation studies in
wireless networks, and to alert the researchers on some common simulation issues and pitfalls.
We conducted a survey on wireless networks, especially on Ad-hoc/Mesh/Sensor/Cognitive
Radio networks studies published in some of the premiere conferences of the wireless networks
from years 2000 to 2008. Table 3 lists the name of all conferences that we considered in our
survey. We only included the full papers on PHY, MAC and Routing layers in our survey, not
the poster and demonstration papers. We reviewed each paper individually avoiding word
searches or other means of automatically gathering results. For consistency, the same person
reviewed all of the papers; to validate the results and to correct the few inconsistencies we had a
second person review all of the papers again.
Average
Confernce
Sr.No. Applicable Area Acceptance Specialized Area Years
Name
Ratio*
ACM
1 Network ≈13% 2000~2008
MobiCom
ACM/IEEE
2 Network/System ≈15% 2000~2008
MobiHoc
3 ACM Sigcomm Network/System ≈15% Ad-hoc/Mesh/Sensor 2000~2008
4 IEEE Infocom Network ≈20% Networks Tracks 2000~2008
5 IEEE Percom Network ≈13% 2003~2008
6 IEEE GlobeCom System ≈40% 2000~2008
7 IEEE WCNC System ≈42% 2000~2008
8 IEEE ICC System ≈34% 2007~2008
9 ACM SenSys Networks/System ≈17% 2003~2008
Sensor Networks
10 EWSN Networks/System ≈16% 2004~2008
IEEE
11 Networks/System ≈35% Cognitive Radio Track 2006~2008
CrownCom
www.intechopen.com
92 Modelling, Simulation and Identiication
*Average Acceptance ratio is calculated over mentioned years, unless otherwise specified.
N/A: we couldn’t provide the average acceptance ratio, as exact figure about the acceptance ratio was not mention on the
respective conference sites.
Table 4 shows the detailed database of survey data; here, we categorized our data into
mainly three categories: MAC layer, Routing layer and PHY layer (especially, for system
simulators). Our database includes all related fields papers from above listed conferences.
From our survey, we come across many simulator tools, and we broadly classified them into
two main categories: “Widely Used” network simulators and “Other” network simulators,
they are summarized in table 5.
ACM
9 2003~2008 - - - 24 - 7 - - - 31
SenSys
10 EWSN 2004~2008 3 - - 12 24 4 - - - 43
IEEE
11 2006~2008 - - - - - - 4 41 50 95
CrownCom
IEEE
12 2005~2008 - - - - - - 1 34 26 61
DySpan
IEEE
13 2008 - - - - - - 1 4 - 5
CogART
www.intechopen.com
Network and System Simulation Tools for Next Generation Networks: A Case Study 93
IEEE
14 2005~2008 - - - - - 14 - 17 8 39
MilCom
Mobile
Networks
15 2000~2008 - - 12 - - - - - - 12
and
Applications
16 MASCOTS 2000~2008 - - 7 - - - - - - 7
ACM
17 2000~2008 - - 8 - - - - - - 8
SigMobile
Total 143 90 87 75 75 104 6 96 132 808
*
2 GloMoSim TOSSIM Monte Carlo Own Simulators
3 J-Sim Monte Carlo - MICA2
4 OMNet++ Own Simulators - Spectrum Analyzer
Simulation Package not Simulation Package
5 OPNet -
mentioned not mentioned
+ Rarely used
6 QualNet Rarely used simulators -
simulators&
*
TOSSIM falls under the category of Emulators.
+
Rarely used simulators: Includes ROSS, JiST/SWAN, Prowler, Emstar, and EmSim, just to name a few. These simulators are
not cited for more then 2/3 research papers in our survey so we put them under the tag named “Rarely used simulators”.
&
Rarely used simulators (System) : Includes Bayesian Estimator, MFC Coding, MDL/AIC, DUALFOIL, SWEET, BLUE-
BCH Estimator, Microwave Studio, BELLHOP, QT Based Simulator, etc. , these simulators are not cited for more then 2/3
research papers in our survey so we put them under the tag named “ Rarely used simulators”.
www.intechopen.com
94 Modelling, Simulation and Identiication
testbed or experimental studies are also gaining popularity in recent years, and their usage
ratio is almost same in Routing and MAC layers. It also shows a good start from the wireless
networks community to present more realistic, practical, and sound research results. But
still there are many issues such as scalability, cost, area, etc., need to be addressed to make
testbed or experimental setup widely accepted among the community. As of current
research practice, simulation is currently the most feasible approach to the quantitative
analysis of wireless networks. As we can see from figure 4 NS-2 is the most popular/used
simulator among the “Widely Used” network simulator tools. To our surprise we find
numerical/mathematical results are more dominating than “Widely Used” tools (NS2 is an
exceptional case), as general trend is to present rigorous simulation results than
mathematically sound results in wireless networks community. It is worth to note that these
numerical results also include the theoretical aspect of the field. From figure 4 we can find a
very interesting observation that in both the layers other/own category is at the top. To
know the reason we further expand survey results on other/ own category as shown in
figure 5.
As we can see from figure 5 major part of other/own category is occupied by the simulator
tools which are not specified in the papers. When the simulator used is not specified within
a published paper, the repeatability and credibility of the simulation study are questionable.
Second topmost category is “own” where researchers have used their self-developed or
custom made simulation tools. It is also difficult, if not possible, to repeat a simulation study
when the simulation is self developed and code is not available. Rest of the simulator tools,
especially Matlab, TOSSIM, Montecarlo, and “rarely used” simulator tools, have a small
portion of participation in wireless networks research. One very important fact come out in
our survey is that a very few papers (hardly 3/4 papers) cited about the code availability,
for whatever reasons but this issue really need an attention from the community. Further
more, we obtained some interesting observations from our survey as shown in figure 6.
The execution and analysis of any experiment/simulation study must be based on
mathematical principles and need to be statistically sound. For any experimental/
simulation study to be statistically sound must present the number of times simulation runs,
confidence levels that exist in the results, and a list of any statistical assumption made. To
our surprise, the large numbers of papers don’t even bother to present this basic information
www.intechopen.com
Network and System Simulation Tools for Next Generation Networks: A Case Study 95
regarding their research results. As we see from figure 6 nearly 150 papers aren’t
independently repeatable because of the lack of simulation’s information. Additionally, the
papers often omitted simulation input parameters such as traffic model or type. As shown
in figure 6 nearly 250 papers didn’t specify any traffic model or type they have used. So, this
lack of basic information raises many questions on the reliability and repeatability of
wireless networks research.
www.intechopen.com
96 Modelling, Simulation and Identiication
layers, we also give our attention on system parameters which play important role to get
reliable and sound results from a simulation as shown in figure 9.
PHY Layer
It is already mentioned that results and analysis getting from a simulation study should be
reliable and acceptable in a range. In general, a model of a physical system has error
associated with its predictions due to the dependence of the physical system’s output
on uncontrollable or unobservable quantities. Confidence level is an important parameter
for the simulation reliability. But from Figure 3 we can see that as like as Routing and MAC
layers, most of the papers didn’t mentioned the traffic models, acceptance levels, and other
statistical parameters explicitly in Physical Layer. It is seen that around 225 papers didn’t
mention any confidence level in their simulation. Again, in more than 236 papers didn’t
mention what traffic model have been used. But in next generation networks traffic
modeling will have to deal with two main issues: the radio resource management scheme
and the effect of the user mobility in the traffic volume per cell. So, information about
traffic model is necessary to further repeatability of a simulation and for reliable output.
Number of independent run of any simulation has also an impact on accurate result. But a
www.intechopen.com
Network and System Simulation Tools for Next Generation Networks: A Case Study 97
large portion of researchers didn’t state any information about that. So, questions may be
raised about the credibility of the simulation based analysis.
To raise the awareness on the lack of reliability, repeatability, and credibility of simulation
based studies we have developed a list of common issues and pitfalls as the starting point
for improvement. We have written the list from our own experiences with simulations as
well as the experience of others in the field. Some common issues and pitfalls are identified
from our survey. We summarize these issues and pitfalls into the following categories:
simulation setup and initial assumptions, simulation execution, and output analysis. They
are summarized with our recommendations in table 6.
www.intechopen.com
98 Modelling, Simulation and Identiication
This paper summarizes the current state of practice, and identified some of the difficult
issues that must be resolved to increase the reliability and credibility of simulation based
studies. Further more, wireless community should take some concrete steps such as
standardization of simulation tools and creating some universal virtual testbeds to resolve
the points of consensus as mentioned above. Universal virtual tesbed could be a very useful
for all the research groups around the globe and can also be used as standard measuring
tool for wireless networks community.
4. Conclusions
In this paper, eight most “widely used” network and system simulators and their strengths
and weaknesses were discussed based on a couple of papers and a survey. Then, the results
of a survey of recent research publications on performance evaluation of networks were
used to show that the majority of results of simulation studies of wireless networks
published in technical literature have many pitfalls/issues. With this paper we documented
these pitfalls and some important issues with some recommendations to increase the
reliability and repeatedly of simulation studies. Finally, we hope, the results presented in
this paper will motivate the researches to put their efforts in thorough descriptions of the
simulation scenarios and taking care of pitfalls in simulation studies of wireless networks.
5. References
I. F. Akyildiz, W. Y. Lee, M.C.Vuran, and S. Mohanty. NeXt generation/dynamic spectrum
access/cognitive radio wireless networks: A survey. Computer Networks 50 (2006)
2127–2159, May 2006.
J. Lessmann, P. Janacik, L. Lachev, and D. Orfanus. Comparative study of wireless network
simulators. in proceeding of ICN, 2008, pp. 517-523.
J. Heidemann, K. Mills, and S. Kumar. Expanding Confidence in Network Simulations. IEEE
Network, vol. 15, no. 5, 2001, pp. 58-63.
S. M. Sanchez. ABC’s of output analysis. in proceeding of the 1999 winter simulator
conference, 1999, pp. 24-32.
B. Schilling. Qualitative comparison of network simulation tools. Technical report, Institute
of Parallel and Distributed Systems (IPVS), University of Stuttgart, January 2005.
www.intechopen.com
Network and System Simulation Tools for Next Generation Networks: A Case Study 99
www.intechopen.com
100 Modelling, Simulation and Identiication
www.intechopen.com
Modelling, Simulation and Identification
Edited by Azah Mohamed
ISBN 978-953-307-136-7
Hard cover, 354 pages
Publisher Sciyo
Published online 18, August, 2010
Published in print edition August, 2010
Modeling, simulation and identification has been actively researched in solving practical engineering problems.
This book presents the wide applications of modeling, simulation and identification in the fields of electrical
engineering, mechanical engineering, civil engineering, computer science and information technology. The
book consists of 17 chapters arranged in an order reflecting multidimensionality of applications related to
power system, wireless communication, image and video processing, control systems, robotics, soil
mechanics, road engineering, mechanical structures and workforce capacity planning. New techniques in
signal processing, adaptive control, non-linear system identification, multi-agent simulation, eigenvalue
analysis, risk assessment, modeling of dynamic systems, finite difference time domain modeling and visual
feedback are also presented. We hope that readers will find the book useful and inspiring by examining the
recent developments in the applications of modeling, simulation and identification.
How to reference
In order to correctly reference this scholarly work, feel free to copy and paste the following:
S. Mehta, Najnin Sulatan and K.S. Kwak (2010). Network and System Simulation Tools for Next Generation
Networks: a Case Study, Modelling, Simulation and Identification, Azah Mohamed (Ed.), ISBN: 978-953-307-
136-7, InTech, Available from: http://www.intechopen.com/books/modelling--simulation-and-
identification/network-and-system-simulation-tools-for-wireless-networks-a-case-study