Learning humanoid–object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion. Human demonstrations provide examples of coordinated interaction, but transferring these behaviors to humanoid robots requires learning how to establish and maintain effective contact forces under different embodiments and dynamics. We present Weave, a unified framework for learning whole-body dexterous humanoid–object interaction from captured human demonstrations. Weave first converts captured human–object interactions into executable robot–object references through contact-aware retargeting and approach-motion completion. At its core is a contact- and geometry-aware policy that jointly controls the humanoid body and articulated fingers across multiple objects and interaction sequences. Evaluation across nine objects yields a 92.5% success rate on trained interactions and, without any additional training, 65.0% on sequences never seen during training. We additionally release ~23 hours of physically executed rollouts, providing robot–object trajectories with contact annotation for downstream interaction-policy learning and physically consistent HOI motion generation.

Weave turns captured human–object interactions into whole-body dexterous loco-manipulation skills in two stages: each demonstration is first converted into an executable robot–object reference, then a single policy learns to track these references in simulation. The four panels below follow one sequence through this pipeline.

• SMPLX Sequences. Each captured demonstration pairs an SMPL-X body motion with the trajectory of the object it manipulates: the grasp and transport the humanoid should reproduce.
• Whole Body Retarget. Whole-body inverse kinematics aligns selected robot links with their SMPL-X counterparts, constraining the pelvis in the horizontal plane and setting its height from ground contact. Since IK captures the gross arm motion but not a stable grasp, a contact-aware hand refinement pulls labeled fingertips onto the object surface, maximizes a force-closure objective, and penalizes penetration.
• Kimodo Completion. The retargeted interaction begins beside the object, so Kimodo synthesizes a locomotion prefix that ends at the initial interaction pose; sampling different approach directions yields several complete references from one clip. Each robot link is then labeled as in contact, neutral, or separated by its distance to the object surface.
• Policy Rollout. A unified contact- and geometry-aware policy is trained with PPO to track these references jointly across all objects and clips. From proprioception, object pose and geometry features, and a short-horizon reference command, it outputs joint-position targets for the body and the proximal finger joints. Rewards combine robot and object tracking with hand-opposition and contact-matching terms; reference state initialization and early termination keep rollouts close to the reference.

We instantiate Weave on a Unitree G1 humanoid with 29 actuated body DoFs and two Inspire dexterous hands with 12 actuated finger DoFs. References are constructed from captured interactions with nine everyday objects and divided into two disjoint splits: a training split of 7,869 trajectories (19.56 h) and a test split of 1,605 trajectories (3.67 h). We report success rate, progress rate, and clip-balanced tracking errors for the pelvis, tracked bodies, joints, and object. The experiments answer three questions:

• Q1. How effectively does the learned policy execute whole-body dexterous loco-manipulation across objects and interaction sequences?
• Q2. How does joint multi-object training compare with single-object specialists in interaction completion and tracking accuracy?
• Q3. How do the policy architecture and optimizer affect learning efficiency?

A single policy learns 7,869 reference trajectories across nine objects, reaching 92.45% success and 96.30% progress on training sequences at 100k iterations. Without additional training, it achieves 64.98% success and 84.52% progress on unseen interaction sequences of the same objects, with comparable tracking errors over executed portions of the trajectories.

Trained jointly on all nine objects for 27k iterations, one policy reaches 95.26% success on ${𝒟}_{\text{test}}$, against 91.48% for nine single-object specialists given the same aggregate budget. It matches the specialists on three objects and improves on the other six, and regresses on none. The specialists nonetheless track more closely on seven of the eight error metrics, so lower imitation error does not imply more reliable interaction completion.

We compare SimBaV2 and MLP backbones under Muon and AdamW within a common budget of 3k iterations on the small-table task. Both Muon runs pull ahead of their AdamW counterparts early and stay ahead, worth 90 to 100 in final episode length, while SimBaV2 adds a further 11 to 21.

@misc{cao2026weave,
  title = {{Weave}: Learning Whole-Body Dexterous Loco-Manipulation
           from Human-Object Interactions},
  author = {Liu Cao and Xingze Wu and Jingzhi Cui and Botian Xu
            and Mingzhi Pei and Ruoqu Chen and Mengdi Xu},
  year = {2026},
  eprint = {2609.16683},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url = {https://arxiv.org/abs/2609.16683}
}