The CPS benchmark compares the performance of different tools when executing a given scenario consisting of a sequence of phases with different cases.

• Tools can be complex configurations (e.g. a given model-to-model transformation and code generator pair).
• Scenarios describe what phases should be performed by the tools and in which order.
• Each phase defines a separate part of the scenario that can be measured with metrics (e.g. runtime or memory usage).
• A case specifies the input model and other aspects that depend on it (e.g. model modifications).

See the TransformationTypes enum for an up-to-date list and this wiki page for additional details.

See the GeneratorTypes enum for an up-to-date list.

This scenario measures the performance of a model-driven application toolchain where an input CPS model is first transformed into a deployment model (model-to-model transformation), then a code generator is executed to output the final source code of the CPS system. After an initial execution, the input model is modified and the toolchain is executed again. This modification and execution loop is executed multiple times.

The scenario has two versions:

• Batch: if the M2M transformation always re-creates the complete output model, then the code generator will also have to be executed on the whole deployment model. In this version, a change monitor that watches the state of the output model would decrease the performance of the transformation.
• Incremental: if the M2M transformation is change aware and only modifies the output model as needed, the code generator can also be executed on the changed parts of the output model. In this version, a change monitor is initialized after the first execution and is used in subsequent code generation phases to update the source code.

Sequence [
  EMFResourceInitialization
  Generation
  Initialization
  M2MTransformation
  M2TTransformation
  Iteration (5 times) [
    Modification
    M2MTransformation (distinguished in results from the first run by adding a postfix)
    M2TTransformation (distinguished in results from the first run by adding a postfix)
  ]
]

Sequence [
  EMFResourceInitialization
  Generation
  Initialization
  M2MTransformation
  M2TTransformation
  ChangeMonitorInitialization
  Iteration (5 times) [
    Modification
    M2MTransformation (distinguished in results from the first run by adding a postfix)
    M2TDeltaTransformation
  ]
]

This scenario is used for comparing the performance of only the model-to-model transformation variants. As before, model modifications are performed to measure the incremental performance as well.

Sequence [
  EMFResourceInitialization
  Generation
  Initialization
  M2MTransformation
  Iteration (5 times) [
    Modification
    M2MTransformation (distinguished in results from the first run by adding a postfix)
  ]
]

The following is an example of the JSON format for defining benchmark configurations:

• Each element of the root benchmark is considered a separate configuration.
• Each configuration is executed by taking the Cartesian product of each array element.
• All elements are currently mandatory!
• For the possible String values, see the corresponding Enums in the code.

The available configurations are found in data.json files in the scripts/configs/config folders.

{
  "benchmark": [
    {
      "scenario" : "TOOLCHAIN_BATCH",
      "cases" : [
        "STATISTICS_BASED"
      ],
      "transformation_types": [
        "BATCH_SIMPLE",
        "BATCH_OPTIMIZED",
        "BATCH_VIATRA_QUERY_RETE",
        "BATCH_VIATRA_QUERY_LOCAL_SEARCH",
        "BATCH_VIATRA_QUERY_LOCAL_SEARCH_NO_FLAT",
        "BATCH_VIATRA_QUERY_LOCAL_SEARCH_DUMB_PLANNER",
        "BATCH_VIATRA_QUERY_LOCAL_SEARCH_WO_INDEXER",
        "BATCH_VIATRA_TRANSFORMATION"
      ],
      "generator_types": [
        "TEMPLATE"
      ],
      "scales": [1,2,4,8,16,32,64],
      "timeout": 1000,
      "runs": 5
    },
    {
      "scenario" : "TOOLCHAIN_INCREMENTAL",
      "cases" : [
        "STATISTICS_BASED"
      ],
      "transformation_types": [
        "INCR_VIATRA_QUERY_RESULT_TRACEABILITY",
        "INCR_VIATRA_EXPLICIT_TRACEABILITY",
        "INCR_VIATRA_AGGREGATED",
        "INCR_VIATRA_TRANSFORMATION"
      ],
      "generator_types": [
        "TEMPLATE"
      ],
      "scales": [1,2,4,8,16,32,64],
      "timeout": 1000,
      "runs": 5
    },
    {
      "scenario" : "TOOLCHAIN_BATCH",
      "cases" : [
        "STATISTICS_BASED"
      ],
      "transformation_types": [
        "BATCH_VIATRA_TRANSFORMATION"
      ],
      "generator_types": [
        "JDT"
      ],
      "scales": [1,2,4,8,16,32,64],
      "timeout": 1000,
      "runs": 5
    },
    {
      "scenario" : "TOOLCHAIN_INCREMENTAL",
      "cases" : [
        "STATISTICS_BASED"
      ],
      "transformation_types": [
        "INCR_VIATRA_AGGREGATED",
        "INCR_VIATRA_TRANSFORMATION"
      ],
      "generator_types": [
        "JDT"
      ],
      "scales": [1,2,4,8,16,32,64],
      "timeout": 1000,
      "runs": 5
    }
  ]
}