Abstract:

An autonomous service robot should be able to interact with its environment safely and robustly without requiring human assistance. Unstructured environments are challenging for robots since the exact prediction of outcomes is not always possible. Even when the robot behaviors are well-designed, the unpredictable nature of physical robot-object interaction may prevent success in object manipulation. Therefore, execution of a manipulation action may result in an undesirable outcome involving accidents or damages to the objects or environment. Situation awareness becomes important in such cases to enable the robot to (i) maintain the integrity of both itself and the environment, (ii) recover from failed tasks in the short term, and (iii) learn to avoid failures in the long term. For this purpose, robot executions should be continuously monitored, and failures should be detected and classified appropriately. In this work, we focus on detecting and classifying both manipulation and post-manipulation phase failures using the same exteroception setup. We cover a diverse set of failure types for primary tabletop manipulation actions. In order to detect these failures, we propose FINO-Net [1] (Failure Is Not an Option), a deep multimodal sensor fusion based classifier network. Proposed network accurately detects and classifies failures from raw sensory data without any prior knowledge. In this work, we use our extended FAILURE dataset [1] with 99 new multimodal manipulation recordings and annotate them with their corresponding failure types. FINO-Net achieves 0.87 failure detection and 0.80 failure classification F1 scores. Experimental results show that proposed architecture is also appropriate for real-time use.

This research is funded by a grant from the Scientific and Technological Research Council of Turkey (TUBITAK), Grant No. 119E-436.

PDF

FAILURE Dataset

FAILURE is a multimodal dataset containing real-world manipulation data recorded with a Baxter robot. There are 5 primitive manipulation actions in the dataset namely: push, pick&place, pour, place-in-container and put-on-top.

Download Data (~9.5GB compressed, ~20GB decompressed)

Download Annotations

Sample executions from the dataset:

	Place	Pour	Put-In	Put-On	Push
Success
Failure

Videos

Bibtex