torte is a declarative experimentation platform for reproducible feature-model analysis research.

Why torte? Take your pick:

• "Tseitin or not Tseitin?" Evaluator
• CNF Transformation Workbench
• KConfig Extractor that Tackles Evolution
• Towards Reproducible Feature-Model Transformation and Extraction
• That's an Obviously Reverse-Engineered Tool Name!
• KConfig = 🍰 config ∧ 🍰 = torte ∎

torte can be used to

• extract feature models from KConfig-based configurable software systems (e.g., the Linux kernel),
• transform feature models between various formats (e.g., FeatureIDE, UVL, and DIMACS), and
• solve feature models with automated reasoners to evaluate the extraction and transformation impact,

all in a fully declarative and reproducible fashion backed by reusable containers. This way, you can

• draft experiments for selected feature models first, then generalize them to a larger corpus later,
• execute experiments on a remote machine without having to bother with technical setup,
• distribute fully-automated reproduction packages when an experiment is ready for publication, and
• adapt and update existing experiments without needing to resort to clone-and-own practices.

torte provides three major setup options:

1. This one-liner will get you started with the default experiment (Docker or Podman required).

   curl -sL https://elias-kuiter.de/torte/ | sh
   

   Choose this option if you do not intend to customize torte.
2. You can also clone this repository and run torte directly.

   git clone --recursive https://github.com/ekuiter/torte.git
   cd torte
   ./torte.sh
   

   Choose this option if piping into a shell is no option for you, or if you want to contribute to torte.
3. A third option is to manually download and set up a release of torte. This increases the total download size, but it also allows you to skip building any Docker images. Choose this option if reproducibility matters a lot, or images fail to build with the other options above.

By default, any of these options will store results in the stages directory. Read on if you want to know more details (e.g., how to execute other experiments).

To run torte, you need:

• an x86_64 or arm64 system ¹ with Linux, macOS, or Windows with WSL
• Git, curl, GNU tools (bash, coreutils, make, grep, and sed)
• Docker (preferably in rootless mode on Linux) or Podman

Experiment files in torte are self-executing - so, you can just create or download an experiment file (e.g., from the experiments directory) and run it.

The following instructions will get you started on a fresh system. By default, each of these instruction sets will install torte into the torte directory. All experiment data will then be stored in the stages directory in your working directory.

# install and set up dependencies
sudo apt-get update
sudo apt-get install -y curl git make uidmap dbus-user-session

# install Docker (see https://docs.docker.com/desktop/install/linux-install/)
curl -fsSL https://get.docker.com | sh
dockerd-rootless-setuptool.sh install

# download and run the default experiment
curl -sL https://elias-kuiter.de/torte/ | sh

# install and set up dependencies (this will replace macOS' built-in bash with a newer version)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
(echo; echo 'eval "$(/opt/homebrew/bin/brew shellenv)"') >> $HOME/.zprofile
eval "$(/opt/homebrew/bin/brew shellenv)"
brew install bash coreutils gnu-sed grep

# install Docker (see https://docs.docker.com/desktop/install/mac-install/)
curl -o Docker.dmg https://desktop.docker.com/mac/main/arm64/149282/Docker.dmg
sudo hdiutil attach Docker.dmg
sudo /Volumes/Docker/Docker.app/Contents/MacOS/install --accept-license
sudo hdiutil detach /Volumes/Docker
rm Docker.dmg
open /Applications/Docker.app

# download and run the default experiment
curl -sL https://elias-kuiter.de/torte/ | sh

# install WSL (see https://learn.microsoft.com/windows/wsl/install)
powershell
wsl --install

# install Docker (see https://docs.docker.com/desktop/install/windows-install/)
Invoke-WebRequest https://desktop.docker.com/win/main/amd64/149282/Docker%20Desktop%20Installer.exe -OutFile Docker.exe
Start-Process Docker.exe -Wait -ArgumentList 'install', '--accept-license'
Remove-Item Docker.exe

# restart your computer, start Docker, then install and set up dependencies
wsl
sudo apt-get update
sudo apt-get install -y curl git make

# download and run the default experiment
curl -sL https://elias-kuiter.de/torte/ | sh

Executing Experiments

• Above, we run the default experiment, which extracts, transforms, and solves the feature model of BusyBox 1.36.0 as a demonstration. To execute another experiment with the one-liner, run curl -sL https://elias-kuiter.de/torte/ | sh -s - <experiment> (information about predefined experiments is available here). You can also write your own experiments by adapting an existing experiment file.
• As an alternative to the self-extracting one-line installer shown above, you can clone this repository and run experiments with ./torte.sh <experiment> (e.g., ./torte.sh busybox-history).
• Yet another alternative is to download a release, which is useful to get a specific version of torte. To ensure reproducibility, each release includes all Docker images as .tar.gz files. These can simply be placed in the root directory of this repository and are then loaded automatically by torte, which sidesteps the image build process.
• A running experiment can be stopped with Ctrl+C. If this does not respond, try Ctrl+Z, then ./torte.sh stop.

Further Tips

• Run ./torte.sh help to get further usage information (e.g., running an experiment over SSH and im-/export of Docker containers).
• Developers are recommended to use ShellCheck to improve code quality.
• If Docker is running in rootless mode, experiments must not be run as sudo. Otherwise, experiments must be run as sudo.
• The first execution of torte can take a while (~30 minutes), as several complex Docker containers need to be built. This can be avoided by loading a reproduction package that includes Docker images (built by ./torte.sh export) or by setting up torte with a release (see above).
• Run PROFILE=y ./torte.sh <experiment> to profile all function calls. This data can be used to draw a flame graph with ./torte.sh (save|open)-speedscope. It can also be used to detect dead code with ./torte.sh detect-dead-code. Note that profiling is enabled at compile time of torte. This means that successive or parallel calls of torte should be run with the same value of PROFILE. This can be ensured easily by running export PROFILE=y once before calling torte.
• Run TEST=y ./torte.sh <experiment> to execute an experiment in test mode (i.e., with a smaller selection of systems). This test mode reduces execution time while maintaining experiment structure and validating toolchain functionality. To run all testable experiments in test mode, run ./torte.sh test (this is also done regularly by GitHub CI).²
• If you prefer Podman over Docker, but both are installed, run FORCE_PODMAN=y ./torte.sh <experiment>. Otherwise, torte will choose whatever is installed (preferring Docker over Podman).
• To remove all Docker artifacts created by torte, run ./torte.sh uninstall. Afterwards, remove the torte directory for full removal (as well as stages and the experiment file if the one-liner was used for setup).

This is a list of all subject systems for which feature-model extraction has been tested and confirmed to work for at least one extraction tool. Other systems or revisions may also be supported.

Detailed system-specific information on potential threats to validity is available in the scripts/systems directory. The files in this directory include templates and convenience functions for working with well-known systems. Most functions extract (an excerpt of) the tagged history of a KConfig feature model. To extract a single revision, you can specify an excerpt with only one commit.

The following tools are bundled with torte and can be used in experiments for extracting and transforming feature models. Most tools are not included in this repository, but cloned and built with tool-specific Docker files in the docker directory. The bundled solvers are listed in a separate table below.

For transparency, we document the changes we make to these tools and known limitations. There are also some general known limitations of torte. ¹²

The following solvers are bundled with torte and can be used in experiments for analyzing feature-model formulas. The bundled solver binaries are available in the docker/solver directory. Solvers are grouped in collections to allow several versions of the same solver to be used.

In addition to the solvers listed below, z3 (already listed above) can be used as a satisfiability and SMT solver.

These #SAT solvers (available here) were used in the evaluations of several papers:

• Evaluating State-of-the-Art #SAT Solvers on Industrial Configuration Spaces (EMSE 2023)
• Tseitin or not Tseitin? The Impact of CNF Transformations on Feature-Model Analyses (ASE 2022)

The #SAT solvers from the collection mcc-2022 should be preferred for new experiments.

These #SAT solvers (available here) were used in the model-counting competition 2022. Not all evaluated solvers are included here, as some solver binaries (i.e., for MTMC and ExactMC) have not been disclosed.

These are miscellaneous solvers from various sources.

A subset of these SAT solvers was used in the evaluation of the paper Tseitin or not Tseitin? The Impact of CNF Transformations on Feature-Model Analyses (ASE 2022). Each solver is the gold medal winner in the main track (SAT+UNSAT) of the SAT competition in the year encoded in its file name. These binaries were obtained from the SAT competition and SAT heritage initiatives. The SAT museum, which has been developed in parallel to this work, also lists and archives single best solvers (SBS). We note differences to the SAT museum where there are any. One overarching difference is that the SAT museum controls for compiler optimizations by patching and compiling every solver from source (whether this is desired depends on the evaluated research question).

These binaries (available here) were obtained from the SAT museum, which was created by Biere et al. As noted above, this solver set is very similar to the sat-competition solver set and has been developed independently and in parallel. The biggest difference is that every solver in the set has been built from source using the same compiler (gcc/g++ 9.4.0), which removes any runtime influence of compiler evolution, allowing for "apple-to-apple comparison" to assess progress in SAT solvers. Biere et al. also made several patches to the original source code of some solvers to make them compatible with modern compilers and fix solver bugs. The sat-competition solver set does not include such restoration efforts and contains solvers "as is".

The experiments directory contains a number of predefined experiments. Some of these experiments are for demo purposes (like the default experiment), while others are used for ongoing or published research. Typically, an experiment consists of an experiment.sh Bash script, which is executed by torte, and an additional evaluation.ipynb Jupyter notebook, which visualizes the experiment's results.

You can also create your own experiments locally. If you want to publish your own experiment, feel free to create a fork or pull request of this repository.

Below, you can find information on the development of torte and its impact so far.

Core contributors:

• Elias Kuiter (University of Magdeburg, Germany)

Further contributors:

• Eric Ketzler (University of Magdeburg, Germany): src/docker/hierarchy
• Urs-Benedict Braun (University of Magdeburg, Germany): experiments/linux-time-travel
• Rami Alfish (University of Magdeburg, Germany): src/docker/configfix
• Lukas Petermann (University of Magdeburg, Germany): torte-dashboard

If you have any feedback, please contact me at kuiter@ovgu.de. New issues, pull requests, or any other kinds of feedback are always welcome.

torte has been used in several research publications:

• Tim Bächle, Erik Hofmayer, Christoph König, Tobias Pett, and Ina Schaefer. Investigating the Effects of T-Wise Interaction Sampling for Vulnerability Discovery in Highly-Configurable Software Systems. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, September 2025.

  Badge: Artifacts Evaluated – Reusable v1.1

• Elias Kuiter, Chico Sundermann, Thomas Thüm, Tobias Heß, Sebastian Krieter, and Gunter Saake. How Configurable is the Linux Kernel? Analyzing Two Decades of Feature-Model History. In Trans. on Software Engineering and Methodology (TOSEM), ACM, April 2025.

  TOSEM'25 Paper | FOSD'24 Slides | TOSEM'25 RCR Report

  Badge: Artifacts Evaluated – Reusable v1.1

• Elias Kuiter, Sebastian Krieter, Chico Sundermann, Thomas Thüm, and Gunter Saake. Tseitin or not Tseitin? The Impact of CNF Transformations on Feature-Model Analyses. In Proc. Int'l Conf. on Automated Software Engineering (ASE). ACM, October 2022.

  ASE'22 Paper | FOSD'22 Slides | ASE'22 Slides | SE'23 Slides | SAT'23 Slides

  Badge: Artifacts Evaluated – Reusable v1.1

This project has evolved through several stages and intends to replace them all:

kmax-vm > feature-model-repository-pipeline > tseitin-or-not-tseitin > torte

• kmax-vm was intended to provide an easy-to-use environment for integrating KClause with PCLocator in a virtual machine using Vagrant/VirtualBox. It is now obsolete due to our Docker integration of KClause.
• feature-model-repository-pipeline extended kmax-vm and could be used to extract feature models from Kconfig-based software systems with KConfigReader and KClause. The results were stored in the feature-model-repository. Its functionality is completely subsumed by torte and more efficient and reliable due to our Docker integration.
• tseitin-or-not-tseitin extended the feature-model-repository-pipeline to allow for transformation and solving of feature models. It was mostly intended as a reproduction package for a single academic paper. Its functionality is almost completely subsumed by torte, which can be used to create reproduction packages for many different experiments.

If you are looking for a curated collection of feature models from various domains, have a look at our feature-model-benchmark.

The source code of this project is released under the LGPL v3 license. To ensure reproducibility, we also provide binaries (e.g., for solvers) in this repository. These binaries have been collected or compiled from public sources. Their usage is subject to each binaries' respective license as specified by the original authors. Please contact me if you perceive any licensing issues.