InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction

Sirui Xu†, Ziyin Wang†, Yu-Xiong Wang‡, Liang-Yan Gui‡,
University of Illinois Urbana-Champaign
†Equal Contribution ‡Equal Advising

NeurIPS 2024

Arxiv    Paper    Demo   



Text-conditioned human motion generation has experienced significant advancements with diffusion models trained on extensive motion capture data and corresponding textual annotations. However, extending such success to 3D dynamic human-object interaction (HOI) generation faces notable challenges, primarily due to the lack of large-scale interaction data and comprehensive descriptions that align with these interactions. This paper takes the initiative and showcases the potential of generating human-object interactions without direct training on text-interaction pair data. Our key insight in achieving this is that interaction semantics and dynamics can be decoupled. Being unable to learn interaction semantics through supervised training, we instead leverage pre-trained large models, synergizing knowledge from a large language model and a text-to-motion model. While such knowledge offers high-level control over interaction semantics, it cannot grasp the intricacies of low-level interaction dynamics. To overcome this issue, we further introduce a world model designed to comprehend simple physics, modeling how human actions influence object motion. By integrating these components, our novel framework, InterDreamer, is able to generate text-aligned 3D HOI sequences in a zero-shot manner. We apply InterDreamer to the BEHAVE and CHAIRS datasets, and our comprehensive experimental analysis demonstrates its capability to generate realistic and coherent interaction sequences that seamlessly align with the text directives.

⚔️ Comparison to Existing Work on Supervised Learning



We evaluate our model on the annotated data available from concurrent work, which is unseen to our zero-shot model, retrieving their generated videos for direct comparison. Remarkably, even without training on these datasets, our method generates high-quality interactions. In addition, our system can accept free-form input from any of their annotations, whereas existing work cannot.



📹 More Visualizations


Contact remains largely unchanged

More challenging cases: Contact is dynamically-changing

📖 Method Overview


InterDreamer can generate vivid 3D human-object interaction sequences guided by textual descriptions. Its zero-shot ability is achieved by integrating semantics and dynamics knowledge from large-scale text-motion data, a large language model (LLM), 3D human-object interaction database, and interaction prior.

  • (i) High-level planning analyzes the description using LLMs and provides guidance to the low-level control.
  • (ii) Low-level control includes a text-to-motion model that translates text into human actions, and an interaction retrieval model for extracting the object's initial pose as the first state.
  • (iii) World model executes the actions and outputs the next state through dynamics modeling. An optimization process is coupled with the dynamics model, projecting the state and action onto valid counterparts.

Failure Cases



BibTex 
@article{xu2024interdreamer,
   title={InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction},
   author={Xu, Sirui and Wang, Ziyin and Wang, Yu-Xiong and Gui, Liang-Yan},
   journal={arXiv preprint arXiv:2403.19652},
   year={2024},
}