Real-world cooking robot system from recipes based on food state recognition using foundation models and PDDL

N Kanazawa, K Kawaharazuka, Y Obinata… - Advanced …, 2024 - Taylor & Francis
Advanced Robotics, 2024Taylor & Francis
Although there is a growing demand for cooking behaviors as one of the expected tasks for
robots, a series of cooking behaviors based on new recipe descriptions by robots in the real
world has not yet been realized. In this study, we propose a robot system that integrates real-
world executable robot cooking behavior planning using the Large Language Model (LLM)
and classical planning of PDDL descriptions, and food ingredient state recognition learning
from a small number of data using the Vision Language model (VLM). We succeeded in …
Although there is a growing demand for cooking behaviors as one of the expected tasks for robots, a series of cooking behaviors based on new recipe descriptions by robots in the real world has not yet been realized. In this study, we propose a robot system that integrates real-world executable robot cooking behavior planning using the Large Language Model (LLM) and classical planning of PDDL descriptions, and food ingredient state recognition learning from a small number of data using the Vision Language model (VLM). We succeeded in experiments in which PR2, a dual-armed wheeled robot, performed cooking from arranged new recipes in a real-world environment, and confirmed the effectiveness of the proposed system.
Taylor & Francis Online