Experiences with the DEVStone benchmark

DEVS is a formal modeling and simulation (M&S) framework that supports hierarchical, modular model composition. DEVS-based M&S environments have been used successfully to understand, analyze, and develop a variety of systems. As the systems under study become more large and complex, performance of the simulator becomes critical. Nevertheless, evaluating the operation of such simulators is a complex process. We present DEVStone, a synthetic benchmark devoted to automate the evaluation of DEVS-based simulators, which generates models with varied structure and behavior. DEVStone was used to study the efficiency of different simulation engines provided by the CD++ toolkit, this results were later compared with similar results on ADEVS with a demanding set of experiments, enabling thorough performance analysis. DEVStone facilitates performance analysis for successive versions (e.g., upgrades or fixes) of the same simulation engine, and provides a common metric to compare different M&S environments.