Evaluating Plan-Property Dependencies: A Web-Based Platform and User Study

Abstract

The trade-offs between different desirable plan properties -- e.g. PDDL temporal plan preferences -- are often difficult to understand. Recent work addresses this by iterative planning with explanations elucidating the dependencies between such plan properties. Users can ask questions of the form ``Why does the plan not satisfy property p?'', which are answered by ``Because then we would have to forego q''. It has been shown that such dependencies can be computed reasonably efficiently. But is this form of explanation actually useful for users? We run a large crowd-worker user study (N=100 in each of 3 domains) evaluating that question. To enable such a study in the first place, we contribute a Web-based platform for iterative planning with explanations, running in standard browsers. Comparing users with vs. without access to the explanations, we find that the explanations enable users to identify better trade-offs between the plan properties, indicating an improved understanding of the planning task.