Object-based image analysis (OBIA) has become very popular since the turn of the century. For high-resolution situations, in particular, where the objects of interest are larger than pixels, methods have been developed that build on image segmentation and on the further classification of objects rather than on pixels. Many studies have shown that OBIA methods are, in principle, more transferable and reapplicable to other images. To obtain comparable results by reapplying a given rule set on (slightly) changed conditions, the rule set must either be able to adapt to the changed conditions or it must be parameterized for manual adaptation. In this context, a rule set can be seen as the more robust the less it has to be changed, and vice versa. In this article we introduce a new method to evaluate the robustness of a rule set. The main assumption is that the amount of necessary adaptations can be measured in conjunction with the quality of classification achieved. We demonstrate that the method introduced is able to (1) evaluate the robustness of a rule set and (2) identify crucial elements of a rule set that need to be reparameterized.