# Temporal Visuo-Tactile Learning for Dexterous Grasp Stability We are pleased to publicly release our dataset, containing 10,000 multi-fingered grasp trials over 200 diverse objects. Each trial records a complete grasp attempt — reaching, hand closure, lifting, and the post-lift outcome — capturing external vision, per-fingertip tactile sensing, and robot proprioception at their native rates on a shared clock, together with a binary grasp stability label. The release contains the HDF5 grasp trials, a set of object reference images, per-object metadata and summary statistics in `dataset.csv`, and two standalone scripts: `data_viewer.py` for browsing and replaying trials, and `resample.py` for building fixed-window, training-ready samples. The scripts, metadata, and their usage instructions are included in `toolkit.zip`. The Download section below explains how to download and extract the release. ## Contributors - Ken Nakahara — Ph.D. student, LASR Lab @ TU Dresden - Aleksei Buvailik — Master's student, LASR Lab @ TU Dresden - Prokhor Kotov — Master's student, LASR Lab @ TU Dresden - Prof. Roberto Calandra — Chair of Machine Learning for Robotics, LASR Lab @ TU Dresden ## License This dataset is licensed under the [Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)](https://creativecommons.org/licenses/by-nc-nd/4.0/). The dataset license covers the HDF5 trials, object reference images, and `dataset.csv`; see `src/data/LICENSE` inside `toolkit.zip`. The Python scripts are licensed separately under the MIT License; see `LICENSE` inside `toolkit.zip`. ## Hardware Setup The data were collected on the following robots and sensors: - Multi-fingered Robotic Hand (4 fingers, 16-DoF): [Tilburg Hand](https://www.tilburg-robotics.eu/) - Robotic Arms (7-DoF): [xArm 7](https://www.ufactory.cc/xarm-collaborative-robot/), the grasping arm fitted with a [6-axis force/torque sensor](https://www.ufactory.us/product/6-axis-force-torque-sensor) - Tactile Sensors (one per fingertip, 4 in total): [Digit 360](https://digit.ml/digit360.html) - RGB-D Camera: [RealSense D435i](https://www.realsenseai.com/products/depth-camera-d435i/) ## Dataset Overview **Collection setup.** Grasps are executed by the arm and the multi-fingered hand listed above. Each of the four fingertips carries a multimodal tactile sensor providing an optical touch camera together with audio, inertial, and pressure sensing. The RGB-D camera is mounted on a second fixed arm and observes a 200 mm x 400 mm tabletop workspace from a side view. **Trial procedure.** In each trial the target object is placed at an arbitrary position in the workspace and localized from the external camera. The arm moves to a randomized grasp pose, sampled by perturbing the detected object position in translation and yaw, and the hand closes toward a sampled 16-DoF target configuration. The robot then lifts the object by 100 mm and holds it. A trial is labeled stable if the object remains in the hand throughout the lift, and unstable otherwise. Every trial is annotated with four phase boundaries — `reach`, `grasp`, `lift`, and `post_lift` — so that any portion of the grasp can be extracted. **Recorded streams.** All streams are recorded asynchronously with their own timestamps, spanning the full attempt — about 12 s per trial, varying with the reaching motion: | Source | Streams | Rate / format | | --- | --- | --- | | External camera | RGB frames, raw depth, intrinsics, extrinsics | 640 x 480, 30 Hz; depth as uint16 millimeters | | Digit 360 (x 4 fingers) | Touch camera frames | 690 x 690, 30 Hz | | Digit 360 (x 4 fingers) | Audio waveform and spectrogram | 48 kHz stereo | | Digit 360 (x 4 fingers) | IMU (accelerometer, gyroscope, magnetometer, quaternion), pressure, temperature | sensor-native rates | | Arm | Joint positions and velocities, 6-D end-effector pose, 6-axis force/torque, commanded actions | 30 Hz | | Hand | 16 joint positions and velocities, commanded actions | 30 Hz | | Grasp | Phase boundaries, sampled grasp positions, pre-grasp and post-lift detection images, stability labels | per trial | **Labels.** Each trial carries a binary stability label in two forms. **The `manual` label is the ground truth**: every trial was visually inspected in its post-lift images and labeled by hand, and this is the label `resample.py` uses. The `automatic` label is the one produced online during collection from the change in object height before and after the lift; it is kept for reference and agrees with the ground truth on 92.34% of trials. Under the ground-truth labels the dataset is near-balanced, with 47.1% stable and 52.9% unstable grasps. **Objects.** The 200 objects span a wide range of physical properties: mass from 1.94 g to 244.4 g (median 32.3 g) and bounding-box volume from 42 cm3 to 1210 cm3 (median 216 cm3). About half are rigid (105) and half deformable (95), across 16 primary material categories, including plastic, silicone, metal, paper, sponge, glass, rubber, wood, textile, and food. The number of trials per object ranges from 14 to 123, with a median of 42. **Per-object metadata.** `dataset.csv` describes every object in one row: the file name of its reference image, the text prompt used to segment the object during collection, primary and secondary materials, a rigid/deformable compliance flag, mass in grams, bounding-box width, depth, and height in millimeters, and the number of trials, the number of successful trials, and the resulting success rate. Object directories, reference images, and the rows of `dataset.csv` share one identifier. The `object_name` column of `dataset.csv` is exactly the directory name and the image file stem, so a row maps directly onto its trials and its picture. ## Download The complete raw dataset is approximately 1.2 TB, so it is distributed as **20 zip archives, each containing 10 object directories**. Download `toolkit.zip`, `data01.zip` through `data20.zip`, and `object_pictures_jpg_square.zip` from the same location as this README. A partial download is fully usable: a subset of the raw-data archives can be downloaded and used on its own if 1.2 TB of storage is not available. Save the downloaded files in one directory. Run all commands in this section from that directory. First, extract the toolkit: ```bash unzip toolkit.zip -d toolkit ``` The data directories are already provided with `.keep` files. Extract the raw-grasp archives into `toolkit/src/data/grasp_data`: ```bash unzip 'data[0-9][0-9].zip' -d toolkit/src/data/grasp_data ``` Each archive expands to object directories directly, producing `toolkit/src/data/grasp_data//.h5`. Verify a complete extraction with: ```bash find toolkit/src/data/grasp_data -mindepth 1 -maxdepth 1 -type d | wc -l # expected: 200 find toolkit/src/data/grasp_data -name '*.h5' | wc -l # expected: 10000 ``` Extract the object reference images into `toolkit/src/data`: ```bash unzip object_pictures_jpg_square.zip -d toolkit/src/data ``` This places the reference images in `toolkit/src/data/object_pictures_jpg_square/`, for example `toolkit/src/data/object_pictures_jpg_square/apple.jpg`. Object metadata is already included in `toolkit/src/data/dataset.csv`. The data are now ready to use. For installation, viewer controls, resampling, and the HDF5 trial format, see `toolkit/src/README.md`. ## Directory Structure After completing the steps above, the directory looks like this. Downloaded ZIP archives and `.keep` files are omitted from the tree. ```text . ├── README.md # this dataset overview └── toolkit/ ├── LICENSE # MIT license for the research code └── src/ ├── README.md # installation, format, and tool usage ├── requirements.txt ├── data_viewer.py ├── resample.py └── data/ ├── LICENSE # dataset license ├── dataset.csv # per-object metadata and success rates ├── grasp_data/ # 200 objects, 10,000 trials total │ ├── / │ │ └── .h5 # one grasp trial │ └── ... ├── object_pictures_jpg_square/ │ ├── .jpg # one reference image per object │ └── ... └── resampled_grasp_data/ # resampler output; initially empty ```