VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Yen-Jen Wang^*,1,2, Jiaman Li^*,‡,1, Sirui Chen^§,1,3, Takara E. Truong^§,1,3, Pei Xu^§,1, Pieter Abbeel^†,1,2, Rocky Duan^†,1, Koushil Sreenath^†,1,2, Angjoo Kanazawa^†,1,2, Carmelo Sferrazza^†,1, Guanya Shi^†,1,4, C. Karen Liu^†,1,3

¹Amazon FAR, ²University of California, Berkeley, ³Stanford University, ⁴Carnegie Mellon University

^*Co-first authors. ^‡Project lead. ^§Equal contribution. ^†Amazon FAR Team Co-Lead.

In Conference on Robot Learning, 2026.

VLK teaser

Abstract

CoRL Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion. Learning this mapping requires synchronized egocentric images, language commands, and robot-compatible kinematic trajectories, yet no existing data source provides this complete tuple at scale. We address this bottleneck by generating vision-language-kinematics (VLK) supervision synthetically in reconstructed scenes. Our pipeline leverages 3D Gaussian Splatting to reconstruct metric-scale indoor environments, synthesizes navigation and object-interaction trajectories using privileged scene information, and renders paired egocentric observations after the fact. We produce 48,000 paired trajectories with no human intervention and train a VLK policy that predicts short-horizon whole-body kinematic trajectories. A whole-body tracker converts these predictions into actions on the physical humanoid. We evaluate on the physical Unitree G1 performing navigation and single-object transport, demonstrating that synthesized interactions in reconstructed scenes provide effective supervision for sim-to-real perception-based humanoid loco-manipulation.

Video

Bibtex

@inproceedings{wang2026vlk,
  title={VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes},
  author={Wang, Yen-Jen and Li, Jiaman and Chen, Sirui and Truong, Takara E. and Xu, Pei and Abbeel, Pieter and Duan, Rocky and Sreenath, Koushil and Kanazawa, Angjoo and Sferrazza, Carmelo and Shi, Guanya and Liu, C. Karen},
  booktitle={Conference on Robot Learning},
  year={2026}
}