🏠 About
This paper addresses a novel task of anticipating 3D human-object interactions (HOIs). Most existing research on HOI synthesis lacks comprehensive whole-body interactions with dynamic objects, e.g., often limited to manipulating small or static objects. Our task is significantly more challenging, as it requires modeling dynamic objects with various shapes, capturing whole-body motion, and ensuring physically valid interactions. To this end, we propose InterDiff, a framework comprising two key steps: (i) interaction diffusion, where we leverage a diffusion model to encode the distribution of future human-object interactions; (ii) interaction correction, where we introduce a physics-informed predictor to correct denoised HOIs in a diffusion step. Our key insight is to inject prior knowledge that the interactions under reference with respect to contact points follow a simple pattern and are easily predictable. Experiments on multiple human-object interaction datasets demonstrate the effectiveness of our method for this task, capable of producing realistic, vivid, and remarkably long-term 3D HOI predictions.
📹 Demo
🎬 Video
🔍 Overview
💡 Key Insight
We present HOI sequences (left), object motions (middle), and objects relative to the contacts after coordinate transformations (right). Our key insight is to inject coordinate transformations into a diffusion model, as the relative motion shows simpler patterns that are easier to predict, e.g., being almost stationary (top), or rotating around a fixed axis (bottom).
🔧 Methodology
Our model contains two parts: the Interaction Diffusion and Interaction Correction modules.
- (i) After each Reverse Diffusion, the Correction Scheduler decides whether or not to perform a correction at that step;
- (ii) If so, we transform the object motion into a contact-based correction reference system and generate a corrected motion through the Interaction Predictor;
- (iii) After that, we transform the prediction back to the ground reference system and inject it into the Forward Diffusion;
🧪 Experimental Results
Qualitative Comparisons with Pure Diffusion on the BEHAVE dataset.
🔗 Citation
InterDiff: Generating 3D Human-Object Interactions with Physics-Informed DiffusionSirui Xu, Zhengyuan Li, Yu-Xiong Wang*, Liang-Yan Gui*
ICCV 2023
BibTex
@inproceedings{
xu2023interdiff,
title={InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion},
author={Xu, Sirui and Li, Zhengyuan and Wang, Yu-Xiong and Gui, Liang-Yan},
booktitle={ICCV},
year={2023},
}