TSC-DL

TSC-DL: Unsupervised Trajectory Segmentation of Multi-Modal Surgical Demonstrations with Deep Learning

Abstract

The growth of robot-assisted minimally invasive surgery has led to sizeable datasets of fixed-camera video and kinematic recordings of surgical subtasks.Temporal segmentation of these trajectories into meaningful contiguous sections is an important first step to facilitate human training and the automation of subtasks. Manual, or supervised, segmentation can be prone to error and impractical for large datasets. We present Transition State Clustering with Deep Learning (TSC-DL), a new unsupervised algorithm that leverages video and kinematic data for task-level segmentation, and finds regions of the visual feature space that mark transition events using features constructed from layers of pre-trained image classification Convolutional Neural Networks (CNNs). We report results on five datasets comparing architectures (AlexNet and VGG), choice of convolutional layer, dimensionality reduction techniques, visual encoding, and the use of Scale Invariant Feature Transforms (SIFT). TSC-DL matches manual annotations with up-to 0.806 Normalized Mutual Information (NMI). We also found that using both kinematics and visual data results in increases of up-to 0.215 NMI compared to using kinematics alone. We also present cases where TSC-DL discovers human annotator errors.

References

• TSC-DL: Unsupervised Trajectory Segmentation of Multi-Modal Surgical Demonstrations with Deep Learning. Adithyavairavan Murali*, Animesh Garg*, Sanjay Krishnan*, Florian T. Pokorny, Pieter Abbeel, Trevor Darrell, Ken Goldberg (* denotes equal contribution).
  IEEE International Conference on Robotics and Automation, (ICRA) 2016
• On Visual Feature Representations for Transition State Learning in Robotic Task Demonstrations. Animesh Garg*, Sanjay Krishnan*, Adithyavairavan Murali, Florian T. Pokorny, Pieter Abbeel, Trevor Darrell, Ken Goldberg (* denotes equal contribution).
  NIPS 2015 Workshop on Feature Extraction 2015.
• Transition State Clustering: Unsupervised Surgical Trajectory Segmentation For Robot Learning. Sanjay Krishnan*, Animesh Garg*, Sachin Patil, Colin Lea, Greg Hager, Pieter Abbeel, Ken Goldberg (* denotes equal contribution).
  International Symposium on Robotics Research (ISRR), 2015.

Authors and Contributors

Animesh Garg, Sanjay Krishnan, Adithya Murali, Florian Pokorny
PIs: Ken Goldberg, Trevor Darrell, Pieter Abbeel

Support or Contact

Please Contact Animesh Garg at animesh.garg@berkeley.edu