Learning Unknown Groundings for Natural Language Interaction with Mobile Robots

Our goal is to enable robots to understand or “ground” natural language instructions in the context of their perceived workspace. Contemporary models learn a probabilistic correspondence between input phrases and semantic concepts (or groundings) such as objects, regions or goals for robot motion derived from the robot’s world model. Crucially, these models assume a fixed and a priori known set of object types as well as phrases and train probable correspondences offline using static language-workspace corpora. Hence, model inference fails when an input command contains unknown phrases or references to novel object types that were not seen during the training. We introduce a probabilistic model that incorporates a notion of unknown groundings and learns a correspondence between an unknown phrase and an unknown object that cannot be classified into known visual categories. Further, we extend the model to “hypothesize” known or unknown object groundings in case the language utterance references an object that exists beyond the robot’s partial view of its workspace. When the grounding for an instruction is unknown or hypothetical, the robot performs exploratory actions to gather new observations and find the referenced objects beyond the current view. Once an unknown grounding is associated with percepts of a new object, the model is adapted and trained online using accrued visual-linguistic observations to reflect the new knowledge gained for interpreting future utterances. We evaluate the model quantitatively using a corpus from a user study and report experiments on a mobile platform in a workspace populated with objects from a standardized dataset. A video of the experimental demonstration is available at: https://youtu.be/XFLNdaUKgW0.

1. 1.

   If \(\lambda _i\) is a noun phrase, the corresponding grounding set \(\varGamma ^i\) contains the objects in the world (i.e., \(\varGamma ^O\)). If \(\lambda _i\) is a prepositional phrase (e.g., “front of a box”) then \(\varGamma ^i\) contains symbols denoting the discretized spatial regions (i.e., front, behind, left etc.) with respect to the objects under consideration (i.e., \(\varGamma ^{RO}\)). If \(\lambda _i\) is a verb phrase referring to the actions that the robot can take (e.g., “move towards a box”, “pick up the block”), then \(\varGamma ^i\) contains the set of constraints defined with respect to pairs of regions (e.g., picking a block can be implicitly expressed as a intersection constraint between the robot’s end-effector and the region occupied by the object).

Authors and Affiliations

1. Massachusetts Institute of Technology, Cambridge, MA, 02139, USA

   Mycal Tucker, Derya Aksaray, Rohan Paul, Gregory J. Stein & Nicholas Roy

Corresponding author

Correspondence to Mycal Tucker .

Editors and Affiliations

1. Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL, USA

   Nancy M. Amato

2. Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA

   Greg Hager

3. Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA

   Shawna Thomas

4. Department of Electrical Engineering, Pontificia Universidad Católica de Chile, Santiago, Chile

   Miguel Torres-Torriti

Reprints and permissions

© 2020 Springer Nature Switzerland AG

Policies and ethics