Sirui Xu  
(徐 思睿)

selected publications

1. 

   InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction

   Sirui Xu, Ziyin Wang, Yu-Xiong Wang, and 1 more author

   In NeurIPS 2024

   Abs arXiv Bib Website

   Text-conditioned human motion generation has experienced significant advancements with diffusion models trained on extensive motion capture data and corresponding textual annotations. However, extending such success to 3D dynamic human-object interaction (HOI) generation faces notable challenges, primarily due to the lack of large-scale interaction data and comprehensive descriptions that align with these interactions. This paper takes the initiative and showcases the potential of generating human-object interactions without direct training on text-interaction pair data. Our key insight in achieving this is that interaction semantics and dynamics can be decoupled. Being unable to learn interaction semantics through supervised training, we instead leverage pre-trained large models, synergizing knowledge from a large language model and a text-to-motion model. While such knowledge offers high-level control over interaction semantics, it cannot grasp the intricacies of low-level interaction dynamics. To overcome this issue, we further introduce a world model designed to comprehend simple physics, modeling how human actions influence object motion. By integrating these components, our novel framework, InterDreamer, is able to generate text-aligned 3D HOI sequences that go beyond existing mocap data without relying on limited pair data. We apply InterDreamer to the BEHAVE, OMOMO, and CHAIRS datasets, and our comprehensive experimental analysis demonstrates its capability to generate realistic and coherent interaction sequences that seamlessly align with the text directives.

   @inproceedings{xu2024interdreamer,
     title = {InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction},
     author = {Xu, Sirui and Wang, Ziyin and Wang, Yu-Xiong and Gui, Liang-Yan},
     booktitle = {NeurIPS},
     year = {2024},
   }

2. 

   InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion

   Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and 1 more author

   In ICCV 2023

   Abs arXiv Bib Code Website

   This paper addresses a novel task of anticipating 3D human-object interactions (HOIs). Most existing research on HOI synthesis lacks comprehensive whole-body interactions with dynamic objects, e.g., often limited to manipulating small or static objects. Our task is significantly more challenging, as it requires modeling dynamic objects with various shapes, capturing whole-body motion, and ensuring physically valid interactions. To this end, we propose InterDiff, a framework comprising two key steps: (i) interaction diffusion, where we leverage a diffusion model to encode the distribution of future human-object interactions; (ii) interaction correction, where we introduce a physics-informed predictor to correct denoised HOIs in a diffusion step. Our key insight is to inject prior knowledge that the interactions under reference with respect to contact points follow a simple pattern and are easily predictable. Experiments on multiple human-object interaction datasets demonstrate the effectiveness of our method for this task, capable of producing realistic, vivid, and remarkably long-term 3D HOI predictions.

   @inproceedings{xu2023interdiff,
     title = {InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion},
     author = {Xu, Sirui and Li, Zhengyuan and Wang, Yu-Xiong and Gui, Liang-Yan},
     booktitle = {ICCV},
     year = {2023},
   }

3. Spotlight

   Stochastic Multi-Person 3D Motion Forecasting

   Sirui Xu, Yu-Xiong Wang, and Liang-Yan Gui

   In ICLR 2023

   Abs arXiv Bib Code Website

   This paper aims to deal with the ignored real-world complexity in prior work on human motion forecasting, emphasizing the social properties of multi-person motion, the diversity of motion and social interaction, and the complexity of articulated motion. To this end, we introduce a novel task of stochastic multi- person 3D motion forecasting. We propose a dual-level generative modeling framework that separately models independent individual movements at the local level and social interactions at the global level. Notably, this dual-level modeling mechanism can be achieved within a shared generative model, through introducing learnable latent codes that represent intents of future movement and switching the codes’ modes of operation at different levels. Our framework is general, and we instantiate it with various multi-person forecasting models. Extensive experiments on CMU-Mocap, MuPoTS-3D, and SoMoF benchmarks show that our approach produces diverse and accurate multi-person predictions, significantly outperforming the state of the art.

   @inproceedings{xu2023stochastic,
     title = {Stochastic Multi-Person 3D Motion Forecasting},
     author = {Xu, Sirui and Wang, Yu-Xiong and Gui, Liang-Yan},
     booktitle = {ICLR},
     year = {2023},
   }

4. Oral

   Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchors

   Sirui Xu, Yu-Xiong Wang, and Liang-Yan Gui

   In ECCV 2022

   Abs arXiv Bib Code Website

   Predicting diverse human motions given a sequence of historical poses has received increasing attention. Despite rapid progress, existing work captures the multi-modal nature of human motions primarily through likelihood-based sampling, where the mode collapse has been widely observed. In this paper, we propose a simple yet effective approach that disentangles randomly sampled codes with a deterministic learnable component named anchors to promote sample precision and diversity. Anchors are further factorized into spatial anchors and temporal anchors, which provide attractively interpretable control over spatial-temporal disparity. In principle, our spatial-temporal anchor-based sampling (STARS) can be applied to different motion predictors. Here we propose an interaction-enhanced spatial-temporal graph convolutional network (IE-STGCN) that encodes prior knowledge of human motions (e.g. spatial locality), and incorporate the anchors into it. Extensive experiments demonstrate that our approach outperforms state of the art in both stochastic and deterministic prediction, suggesting it as a unified framework for modeling human motions.

   @inproceedings{xu22stars,
     title = {Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchors},
     author = {Xu, Sirui and Wang, Yu-Xiong and Gui, Liang-Yan},
     booktitle = {ECCV},
     year = {2022},
   }