HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning

Despite the importance of tactile sensing for reliable manipulation, most existing Vision-Language-Action (VLA) datasets remain vision-only, and those that do incorporate tactile information typically lack the joint combination of task diversity, language conditioning, and action trajectories. Furthermore, existing teleoperation pipelines rarely provide haptic feedback to the operator, despite its established role in demonstration quality and manipulation stability.

In this work, we present HapTile, a contact-grounded visuotactile manipulation dataset that advances beyond vision-only trajectory datasets by embedding physical interaction sensing at two levels: fingertip tactile feedback at the robot end-effector, and haptic-informed demonstrations at the teleoperator side. The data collection platform integrates haptic feedback directly into the teleoperation controller, enabling the operator to perceive contact interactions in real time. It is built around a standard and reproducible robotic system equipped with custom-designed fingertip tactile sensors.

The dataset comprises everyday manipulation tasks spanning a broad range of contact-rich skills, including pick-and-place, folding, pressing, stacking, and other routine activities. Each task is paired with language instructions that condition the policy on the manipulation objective, together with synchronized visuotactile observations and action trajectories. In addition, we provide a benchmarking study on contact-rich policy learning using two baseline models to evaluate the effectiveness of the proposed contact-grounded dataset.

Each task folder (e.g., fold_Tshirt/) contains multiple demonstrations (named by date-time). Note that attrs, frames/, and videos/ are not real directories — they are HDF5 groups/datasets inside trajectory.h5, accessed in Python via h5py, e.g. trajectory["frames/joint_positions"].

        task/
        ├── 0612_143005/                  # demonstration folder (date-time)
        │   ├── freq.txt                               # recording frequency metadata
        │   └── trajectory.h5                          # HDF5 file
        │       ├── attrs                              # top-level attributes (episode metadata)
        │       │   ├── frame_count                     # total number of frames in the episode
        │       │   ├── video_fps                       # recording frame rate 
        │       │   └── timestamps                      # ISO 8601 datetime string per frame
        │       ├── frames/                            # HDF5 group: time-series robot state and tactile data
        │       │   ├── joint_positions                # 7-DOF joint angles (radians)
        │       │   ├── ee_pos_quat                    # end-effector position + orientation 
        │       │   ├── gripper_position                # gripper opening, normalised [0–1]
        │       │   ├── control                        # actions: dims 0–5 = 6-DOF delta pose, dim 6 = gripper
        │       │   ├── tactile_left_marker_motion      # scalar motion magnitude from left tactile (recorded live)
        │       │   └── tactile_right_marker_motion     # scalar motion magnitude from right tactile (recorded live)
        │       ├── language_instructions               # HDF5 group: language instructions for the task
        │       └── videos/                            # HDF5 group: multi-camera video streams (raw compressed bytes)
        │           ├── base_camera_rgb_0               # wrist camera RGB stream — raw MP4 bytes
        │           ├── base_camera_rgb_1               # front camera RGB stream — raw MP4 bytes
        │           ├── base_camera_depth_0             # depth camera 0 stream — raw MP4 bytes
        │           ├── base_camera_depth_1             # Depth camera 1 stream — raw MP4 bytes
        │           ├── tactile_left_rgb                # left tactile RGB stream — raw MP4 bytes
        │           └── tactile_right_rgb               # right tactile RGB stream — raw MP4 bytes
        ├── 0612_150241/
        │   ├── freq.txt
        │   └── trajectory.h5                          # same internal structure as above
        │
        ⋮
        │
        └── 0612_204719/
            ├── freq.txt
            └── trajectory.h5                          # same internal structure as above

The dataset is hosted on HuggingFace. To download, extract and visualize the dataset, you can use the following command lines:

            # Clone the repository
            git clone https://github.com/HapTile-dataset/Dataset_quickstart.git
            cd Dataset_quickstart

            # Install dependencies
            pip install -r requirements.txt

            # Run the quickstart script
            python quickstart.py --task fold_Tshirt.zip --frame 150

RGB tactile images from both fingers are captured at every control step. Contact induces characteristic image changes and marker displacements that encode contact geometry and task-specific interaction forces. We track these marker displacements online. Displacement relative to the initial reference is converted into a scalar motion score that approximates contact-induced deformation and slip. This tactile motion signal is stored alongside RGB observations and action trajectories in the dataset, and is also used to drive real-time haptic feedback during teleoperation.

To close the perception–action loop during teleoperation, tactile contact information is fed back to the operator as real-time haptic vibration through the Meta Quest controller. The tactile motion score obtained from marker tracking algorithm is normalised and discretely mapped to vibration amplitude, enabling the operator to feel when sufficient grip force is established.

Data collection follows a structured protocol to promote policy generalisation and language grounding:

• Randomised object poses and target positions within the workspace, with fixed end-effector reset between episodes
• Scene diversity via one or two clutter objects added per scene
• Two language-specified target states per task, enabling generalisation splits that verify the policy conditions on language rather than converging to a single outcome

We evaluate two policy families — a Diffusion Policy baseline and fine-tuned π₀​ — on four contact-rich tasks (bottle turning, whiteboard wiping, liquid pouring, peg insertion) across three modality configurations: V-only. RGB views + proprioception, no tactile; V+T. adds raw tactile images from both fingers; V+TM. adds marker displacement features for contact geometry and slip.