Tyler Lum
@tylerlum23
CS PhD student @Stanford | Robotics & AI | Prev NVIDIA, Tesla, UBC
馃馃 Introducing Human2Sim2Robot!聽 馃挭馃 Learn robust dexterous manipulation policies from just one human RGB-D video. Our Real鈫扴im鈫扲eal framework crosses the human-robot embodiment gap using RL in simulation. #Robotics #DexterousManipulation #Sim2Real 馃У1/7
Boston Dynamics collaborated with NVIDIA to demonstrate DextrAH-RGB, a workflow for dexterous grasping from stereo RGB input. The end-to-end policy for Atlas robot, trained entirely in NVIDIA Isaac Lab, transfers zero-shot from simulation to the real robot.
The Dex team at NVIDIA is defining the bleeding edge of sim2real dexterity. Take a look below 馃У There's a lot happening at NVIDIA in robotics, and we鈥檙e looking for good people! Reach out if you're interested. We have some big things brewing (and scaling :)
We find keypoint trajectories to be a powerful interface between VLM planning & RL control VLM: Generates object + hand motion plan from a task prompt & RGB-D image (perception + commonsense) RL policy: Conditioned on the plan, learns low-level dexterous control (0-shot sim2real)
Can we teach dexterous robot hands manipulation without human demos or hand-crafted rewards? Our key insight: Use Vision-Language Models (VLMs) to scaffold coarse motion plans, then train an RL agent to execute them with 3D keypoints as the interface. 1/7
Teleoperation is slow, expensive, and difficult to scale. So how can we train our robots instead? Introducing X-Sim: a real-to-sim-to-real framework that trains image-based policies 1) learned entirely in simulation 2) using rewards from human videos. portal-cornell.github.io/X-Sim