This file was generated on 2026-05-16 by Tim Engelbracht A GENERAL INFORMATION 1. Title of the dataset: Hoi!-A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation 2. Brief description of the research project and its aims: We present a dataset for force-grounded, cross-view articulated manipulation that couples what is seen with what is done and what is felt during real human interaction. The dataset contains 3048 sequences across 381 articulated objects in 38 environments. Each object is operated under four embodiments - (i) human hand, (ii) human hand with a wrist-mounted camera, (iii) handheld UMI gripper, and (iv) a custom Hoi! gripper - where the tool embodiment provides synchronized end-effector forces and tactile sensing. Our dataset offers a holistic view of interaction understanding from video, enabling researchers to evaluate how well methods transfer between human and robotic viewpoints, but also investigate underexplored modalities such as force sensing and prediction. Further information can be found on the Website. 3. Author Information A. Investigator Contact Information Name: Tim Engelbracht Institution: ETH Zürich Email: tengelbracht@ethz.ch Name: René Zurbrügg Institution: ETH Zürich Email: zrene@ethz.ch Name: Matteo Wohlrapp Institution: Technical University Munich Email: mwohlrapp@student.ethz.ch Name: Martin Büchner Institution: University of Freiburg Email: buechner@cs.uni-freiburg.de Name: Abhinav Valada Institution: University of Freiburg Email: valada@cs.uni-freiburg.de Name: Marc Pollefeys Institution: ETH Zürich Email: marc.pollefeys@inf.ethz.ch Name: Hermann Blum Institution: University of Bonn Email: blumh@uni-bonn.de B. Project Supervisor Contact Information Name: Zuria Bauer Institution: ETH Zürich Email: zbauer@ethz.ch C. In case of questions related to this dataset, please contact: Name: Tim Engelbracht Institution: ETH Zürich Email: tengelbracht@ethz.ch 4. Date of data collection: 2025-06-01 to 2026-02-01 5. Information about funding sources that supported the collection of the data: ETHAICenter, ETH Foundation Project 2025-FS-352, the SNSF Advanced Grant 216260, and the Lamarr Institute for Machine Learning and Artificial Intelligence. It was further supported by the Robotics Institute Germany, Google, and Meta. 6. Language of the dataset: English 7. Geographic location of data collection: Switzerland, Zürich B DATA & FILE OVERVIEW 1. File List: ``` / ├── gripper/ │ ├── interaction_splitting_info_.json │ ├── interaction_splitting_info__confirmed.json │ ├── aria_gripper/ │ │ └── __gripper_vrs/ │ │ ├── anonymization.json │ │ ├── statistics.json │ │ ├── time_sync_info.json │ │ ├── anonymization_cache/ │ │ ├── calib/ │ │ ├── camera_depth/ │ │ ├── camera_rgb/ │ │ ├── eye_gaze/ │ │ ├── hand_tracking/ │ │ ├── slam/ │ │ └── visual_registration/ │ ├── aria_human/ │ │ └── __gripper_vrs/ │ │ └── (same general Aria structure as above) │ ├── gripper/ │ │ └── __gripper_bag/ │ │ ├── anonymization.json │ │ ├── time_sync_info.json │ │ ├── anonymization_cache/ │ │ ├── calib/ │ │ ├── digit/ │ │ ├── dynamixel_workbench/ │ │ ├── force_torque/ │ │ ├── gripper_force_trigger/ │ │ ├── tf_static/ │ │ └── zedm/ │ ├── iphone_1/ or iphone_1 ()/ │ │ └── __gripper/ │ │ ├── anonymization.json │ │ ├── metadata (contains camera intrinsics of the corresponding iphone) │ │ ├── statistics.json │ │ ├── time_sync_info.json │ │ ├── anonymization_cache/ │ │ ├── camera_depth/ │ │ ├── camera_rgb/ │ │ ├── poses/ │ │ ├── rgbd/ │ │ ├── visual_registration/ │ │ └── poses_aligned/ (if present) │ └── iphone_2/ or iphone_2 ()/ │ └── (same general iPhone structure as iphone_1) ├── hand/ │ ├── interaction_splitting_info_.json │ ├── interaction_splitting_info__confirmed.json │ ├── aria_human/ │ │ └── __hand_vrs/ │ │ └── (same general Aria structure; multi_slam/ may also be present) │ ├── iphone_1/ or iphone_1 ()/ │ │ └── __hand/ │ │ └── (same general iPhone structure; poses_aligned/ may also be present) │ └── iphone_2/ or iphone_2 ()/ │ └── (same general iPhone structure as iphone_1) ├── leica/ │ ├── .json │ ├── / │ │ ├── images/ │ │ ├── points/ │ │ ├── points_downsampled/ │ │ ├── mesh/ (if present) │ │ ├── pano_tiles/ (if present) │ │ ├── instance_annotations/ (if present) │ │ └── instance_annotations_3d/ (if present) │ └── / ├── umi/ │ ├── interaction_splitting_info_.json │ ├── interaction_splitting_info__confirmed.json │ ├── aria_human/ │ │ └── __umi_vrs/ │ │ └── (same general Aria structure; multi_slam/ may also be present) │ ├── iphone_1/ │ │ └── __umi/ │ │ └── (same general iPhone structure as above) │ ├── iphone_2/ │ │ └── __umi/ │ │ └── (same general iPhone structure as above) │ └── umi_gripper/ │ └── __umi/ │ ├── anonymization.json │ ├── time_sync_info.json │ ├── anonymization_cache/ │ ├── calib/ │ ├── camera_rgb/ │ ├── odometry/ │ ├── slam/ │ ├── telemetry/ │ └── visual_registration/ └── wrist/ ├── interaction_splitting_info_.json ├── interaction_splitting_info__confirmed.json ├── aria_human/ │ └── __wrist_vrs/ │ └── (same general Aria structure; multi_slam/ may also be present) ├── aria_wrist/ │ └── __wrist_vrs/ │ └── (same general Aria structure as above) ├── iphone_1/ │ └── __wrist/ │ └── (same general iPhone structure as above) └── iphone_2/ └── __wrist/ └── (same general iPhone structure as above) ``` The data may server a multitude of different research directions, such as multimodal articulation understanding, visual and tactile force prediction, multi-view computer vision, multimodal robot learning, amongst others. Each recording location (First folder level) may contain a number of articulated parts, that were interacted with in a recording session using one of four manipulation conditions: A Custom Hoi! Gripper, UMI gripper, human hand with wrist camera or human hand (second folder level). Each recording session contains numerous spatially and temporally aligned recording modules, such as the manipulation module, egocentric Aria module and exocentric iPhone module (third level). Each module carries module-specific data streams, such as RGB-D streams or slam trajectories and calibration information (forth level). Additionally, we provide session-level interaction splitting annotations. Lastly, we provide Leica laser scans, including 3D axis annotations and 3D object instance annotations. metadata files in the iPhone folders contain device-specific calibrations and are distinct given there filepath (e.g. iphone_1/metadata contains meta data (calibration info) for the corresponding recording device iphone_1, NOT for iphone_2.) Further Information can be found on the project website: https://timengelbracht.github.io/Hoi-Dataset-Website/ 2. Are there multiple versions of the dataset? No C SHARING/ACCESS INFORMATION 1. Was data derived from another source?: No 2. Licenses/restrictions placed on the data: CC-BY 3. Links to publications that cite or use the data: https://doi.org/10.48550/arXiv.2512.04884 (Also presented at CVPR'26, proceedings not available yet) 4. Links to other publicly accessible locations of the data: https://timengelbracht.github.io/Hoi-Dataset-Website/ 5. Links/relationships to ancillary datasets: n/a D METHODOLOGICAL INFORMATION 1. Description of methods used for collection/generation of data: Recorded furniture interactions with Custom-built Hoi gripper, UMI gripper, Meta Aria glasses, iPhones, Leica RTC360 laser scanner. 2. Methods for processing the data: Custom Hoi data processing pipeline. Temporal alignment via QR-tag detection, spatial alignment via visual localization, code can be found on the project website. 3. Instrument- and/or software-specific information needed to interpret the data: Open-source Python libraries (example: Open3D, OpenCV) 4. People involved in sample collection, processing, analysis and/or submission: See Authors, Tiffany Portela (Data Recording), Kavya Shankar (Data Annotation), James Ford, Jakob Engel, Mingfei Yan and Richard Newcombe (various other contributions) 5. Describe any quality-assurance procedures performed on the data: Anonymization via EgoBlur Software (https://github.com/facebookresearch/EgoBlur)