# Learning Gentle Grasping Using Vision, Sound, and Touch

We are pleased to open-source our dataset, which contains 1,500 robotic grasps collected for the paper [Learning Gentle Grasping Using Vision, Sound, and Touch](https://lasr.org/research/gentle-grasping).

Additionally, we provide scripts to visualize the data and process raw data into a training dataset for a PyTorch model used in this work.

## Contributors
- **Ken Nakahara** – Ph.D. student / [LASR Lab](https://lasr.org) @ TU Dresden
- **[Prof. Roberto Calandra](https://www.robertocalandra.com/about/)** – Chair of Machine Learning for Robotics / [LASR Lab](https://lasr.org) @ TU Dresden

## Hardware Setup
The experimental environment consists of the following robots and sensors:
- **Multi-fingered Robotic Hand (16-DoF):** [Allegro Hand v4.0](http://wiki.wonikrobotics.com/AllegroHandWiki/index.php/Allegro_Hand_v4.0)
- **Robotic Arms (7-DoF):** [xArm 7](https://www.ufactory.cc/xarm-collaborative-robot/)
- **Tactile Sensors:** [DIGIT](https://digit.ml/digit.html)
- **RGB-D Camera:** [Intel RealSense D435i](https://www.intelrealsense.com/depth-camera-d435i/)
- **Audio Sensor:** A commodity microphone

## Collected Data
The dataset is stored under the "collected_data" folder, which contains raw data from 1,500 grasping trials.

### Grasping Phases
Each grasping trial consists of four phases, recorded at specific timestamps (t_*):
- **t_0**: Before grasping
- **t_1**: Initial grasping
    - **t_12**: After releasing the initial grasp (for data augmentation)
- **t_2**: Regrasping
- **t_3**: Lifting 

### Data Captured per Phase
For each phase at time t_*, we collected:
- **Visuo-tactile observations** (t_0, 1, 12, 2, 3)
    - **RGB Images** from RealSense: `camera_rgb_*.png`
    - **Depth Heat Maps** from RealSense: `camera_depth_*.png`
    - **Tactile Images** from DIGIT Sensors:
        - Thumb: `touch_thumb_*.png`
        - Middle Finger: `touch_middle_*.png`
- **Actions** (t_0, 1):
    - Hand joint angles (16-DoF): `action*_hand.npy`
    - Arm joint angles (7-DoF): `action*_arm.npy`
    - End-effector's pose shift (dx, dy, dz, dyaw): `action1_regrasp_pose.npy`
- **Observed Hand Joint Angles**:
    - Before regrasp: `hand_obs_before.npy`
    - After regrasp: `hand_obs_after.npy`
- **Audio Recordings** during regrasping (3s): `record_s2.wav`
- **Labels** (i.e., Grasp Outcomes):
    - Self-supervised labels (stability: 1/0, gentleness: 1/0): `labels.npy`
    - Supervised labels (for grasp stability via our visual inspection): `labels_supervised.npy`

## Supplemental Scripts
We provide scripts to view the raw dataset and process it for training. To execute these scripts, install the required dependencies.

### Installation
_(Tip: Use a virtual environment such as Conda)_
```bash
conda create -n test python=3.8
conda activate test
cd dataset_gentle_grasping
pip install -r requirements.txt
```

### Scripts
- **`view_data.py`** – Script to visualize the dataset
    ```bash
    python view_data.py collected_data/
    ```
- **`process_data.py`** –  Script to process the raw data into a training dataset for a PyTorch model. (**Caution**: the processed dataset (with 9,000 samples by default) will be saved as a pth file with ~11GB of data in the same folder as this script.)
    ```bash
    python process_data.py
    ```

## Contact
For any questions, feel free to reach out to us at [knakahara@lasr.org](mailto:knakahara@lasr.org).