Lihan Zha

Want your imitation learning policy to generalize better, but how to collect data to achieve this? 🤖🤔 Enter Factored Scaling Curves (FSC): a tool that quantifies how policy success scales with demos for each environmental factor, enabling principled data collection 📈 . 🌐…

Tactile interaction in the wild can unlock fine-grained manipulation! 🌿🤖✋ We built a portable handheld tactile gripper that enables large-scale visuo-tactile data collection in real-world settings. By pretraining on this data, we bridge vision and touch—allowing robots to:…

I converted one of my favorite talks I've given over the past year into a blog post. "On the Tradeoffs of SSMs and Transformers" (or: tokens are bullshit) In a few days, we'll release what I believe is the next major advance for architectures.

Join us at two workshops #RSS2025 on 6/25! 📍 Benchmarking Robot Manipulation: Improving Interoperability and Modularity (RTH 526, poster stand 88) 📷 Oral talk: 10.30-10.35 📍 Large Foundation Models for Interactive Robot Learning (SGM 123, poster stand 135) 📷 Lightning…

🚀 Excited to introduce SAFE, our work on multitask failure detection for Vision-Language-Action (VLA) models! 🔍 SAFE is a simple yet powerful detector that leans from VLAs’ semantic-rich internal feature space and outputs a scalar score indicating the likelihood of task failure

Imitation learning is not merely collecting large-scale demonstration data. It requires effective data collection and curation. FSC is a great example of this! Join Lihan’s session and chat with him to learn how to make your policy more general from a data-centric perspective!

Check out our recent work, Particle-Grid Neural Dynamics, led by the brilliant @kaiwynd , on 3D world modeling for diverse and challenging deformable objects 🧵👕🧸🛍️📦🍞!!!! The key insight is the hybrid Particle+Grid representation, enabling PGND to capture both fine-grained…

Join us at two workshops #RSS2025 on 6/21! 📍 Resource Constrained Robotics (RTH109) 🗣️ Oral talk: 11:00–11:15 📍 Continual Robot Learning from Humans (OHE132) 🖼️ Spotlight poster: 10:30–11:00 Come by and chat—we’re excited to share our work!

Normally, changing robot policy behavior means changing its weights or relying on a goal-conditioned policy. What if there was another way? Check out DynaGuide, a novel policy steering approach that works on any pretrained diffusion policy. dynaguide.github.io 🧵

Excited to share more of what I've been working on for while now! We've trained a world model to solve policy evaluation, where we see enough correlation to real-world evaluations for practical production usage.

(1/n) Time to unify your favorite visual generative models, VLMs, and simulators for controllable visual generation—Introducing a Product of Experts (PoE) framework for inference-time knowledge composition from heterogeneous models.

Robots struggle to find objects like humans. Why? Understanding "behind the box" (semantics) isn't enough – they need to plan precise, efficient actions to get there. Key Insight: VLMs propose where to look ("Maybe behind the box?"). World models evaluate VLM proposals and…

How can we move beyond static-arm lab setups and learn robot policies in our messy homes? We introduce HoMeR, an imitation learning agent for in-the-wild mobile manipulation. 🧵1/8

Check out a recent talk I gave about how we can use structured world models + inference/planning to construct embodied agents that can generalize to many unseen tasks!

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

Introducing Mobi-π: Mobilizing Your Robot Learning Policy. Our method: ✈️ enables flexible mobile skill chaining 🪶 without requiring additional policy training data 🏠 while scaling to unseen scenes 🧵↓

How can we embed human-like reasoning into driving policies, without having to incur extra inference time? verdi-driving.github.io VERDI distills the reasoning process and commonsense knowledge from a VLM into an e2e model, boosting performance while maintaining inference speed!

Great new paper by @LihanZha, @ApurvaBadithela, @mzhangio, @justinlidard, @Majumdar_Ani, @allenzren, @shahdhruv_, and team. As we scale data collection for robotics, it must be done intelligently by choosing which data variations to focus resources on. This paper provides a…

Nice to see the use of ManiSkill3 in this work! Simulation is not just useful for RL training. It provides some good cheap deterministic test beds, perfect for testing imitation learning scaling laws at scale. Years of data in hours

Happy to share recent work on Factored Scaling Curves for Guiding Data Collection led by my colleague @LihanZha in collaboration with @allenzren @shahdhruv_ @Majumdar_Ani @justinlidard and others! x.com/LihanZha/statu…

Data really matters for policy generalizing to diverse environment conditions as seen in pi05 etc. In this work we find strong correlation b/w # data and test performance, enabling targeted data collection. We also find offline metric (policy embedding) to give strong signal!