Temporal Visuo-Tactile Learning for Dexterous Grasp Stability

Nakahara, Ken; Buvailik, Aleksei; Kotov, Prokhor; Calandra, Roberto

Temporal Visuo-Tactile Learning for Dexterous Grasp Stability

Type of the data datacite.resourceTypeGeneral	Dataset
Total size of the dataset datacite.size	1078004133457
Author dc.contributor.author	Nakahara, Ken
Author dc.contributor.author	Buvailik, Aleksei
Author dc.contributor.author	Kotov, Prokhor
Author dc.contributor.author	Calandra, Roberto
Upload date dc.date.accessioned	2026-10-08T15:12:37Z
Publication date dc.date.available	2026-10-08T15:12:37Z
Data of data creation dc.date.created	2026
Publication date dc.date.issued	2026-10-08
Abstract of the dataset dc.description.abstract	This dataset contains 10,000 multimodal robotic grasp trials across 200 everyday objects with diverse shapes, sizes, materials, and compliance, collected for the paper Temporal Visuo-Tactile Learning for Dexterous Grasp Stability. The platform combines a 7-DoF xArm7, a four-fingered, 16-DoF Tilburg Hand with a Digit 360 tactile sensor on each fingertip, and an external RealSense D435i RGB-D camera. Each trial continuously records external RGB-D observations, arm and hand proprioception, tactile images, audio, inertial measurements, pressure, and commanded actions throughout the full grasp attempt, from reaching and hand closure to lifting and post-lift outcome checking. The recordings include phase annotations and binary grasp-stability labels indicating whether the object was successfully lifted and held for three seconds without dropping. Timestamped, asynchronously recorded observations and actions, together with phase start times, allow selected intervals to be resampled into synchronized multimodal sequences; the pre-lift window used in the paper is one such example. The release also provides object reference images, per-object metadata, and Python tools for data visualization and resampling into training-ready samples. These full-grasp recordings support reuse beyond the prediction setting studied in the paper, including action-conditional grasp stability prediction, contact dynamics modeling, and multimodal representation learning for dexterous manipulation.
Public reference to this page dc.identifier.uri	https://opara.zih.tu-dresden.de/handle/123456789/2987
Public reference to this page dc.identifier.uri	https://doi.org/10.25532/OPARA-1588
Publisher dc.publisher	Technische Universität Dresden
Licence dc.rights	Attribution-NonCommercial-NoDerivatives 4.0 International	en
URI of the licence text dc.rights.uri	http://creativecommons.org/licenses/by-nc-nd/4.0/
Specification of the discipline(s) dc.subject.classification	4::44::409::409-05
Title of the dataset dc.title	Temporal Visuo-Tactile Learning for Dexterous Grasp Stability
Software opara.descriptionSoftware.ResourceProcessing	Python
Software opara.descriptionSoftware.ResourceViewing	Python
Project abstract opara.project.description	Humans can grasp everyday objects with almost perfect success rates using fingertip tactile feedback, yet much of the robotic grasping literature emphasizes vision-based grasp selection with parallel grippers. In this work, we systematically investigate how high-resolution, dynamic tactile sensing contributes to grasp stability prediction and model-guided grasping in dexterous robotic hands. To this end, we collected a dataset of 10,000 grasp trials across 200 objects using a multi-fingered robotic hand equipped with four Digit 360 tactile sensors, recording external vision, proprioception, and tactile streams throughout each grasp. With this dataset, we trained end-to-end temporal multimodal models to predict post-lift stability from pre-lift grasp observations and compared sensing modalities and encoding backbones. Experimental results and controlled input ablations show that incorporating touch, and particularly high-resolution, dynamic touch, improves grasp stability prediction. Finally, we deployed the learned predictor as an online stability gate on the real robot, where visuo-tactile model-guided regrasping improved the success rate among executed lifts by 10.5 percentage points over a non-tactile gate. These results show how rich fingertip sensing and expressive temporal models that capture the dynamics of touch can support learned grasping with multi-fingered hands without explicit contact or force modeling, providing a scalable data-driven path from tactile experience toward stable dexterous manipulation.
Funding Acknowledgement opara.project.fundingAcknowledgement	This work was partly supported by the project GeniusRobot (01IS24083) funded by the Federal Ministry of Research, Technology and Space (BMFTR), by BMFTR in the DAAD project 57616814 (SECAI, School of Embedded Composite AI), by BMFTR under the Robotics Institute Germany (RIG), and by the German Research Foundation (DFG) under the Cluster of Excellence CARE: Climate-Neutral And Resource-Efficient Construction (EXC 3115, project number 533767731).
Public project website(s) opara.project.publicReference	https://lasr-lab.github.io/dexterous-grasp-stability/
Project title opara.project.title	Temporal Visuo-Tactile Learning for Dexterous Grasp Stability