Joel Jang

Introducing 𝐃𝐫𝐞𝐚𝐦𝐆𝐞𝐧! We got humanoid robots to perform totally new 𝑣𝑒𝑟𝑏𝑠 in new environments through video world models. We believe video world models will solve the data problem in robotics. Bringing the paradigm of scaling human hours to GPU hours. Quick 🧵

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to…

I've been a bit quiet on X recently. The past year has been a transformational experience. Grok-4 and Kimi K2 are awesome, but the world of robotics is a wondrous wild west. It feels like NLP in 2018 when GPT-1 was published, along with BERT and a thousand other flowers that…

Compete for a $560,000 Prize Pool at IROS 2025 AgiBot World Challenge! 💰 The AgiBot World Challenge – Manipulation Track is LIVE! Hosted by @AgiBot and @OpenDriveLab at #IROS2025. 🚀 Challenge: Tackle 10 complex Sim2Real manipulation tasks. 🛠️ Resources: Access a unique…

Check out Cosmos-Predict2, a new SOTA video world model trained specifically for Physical AI (powering GR00T Dreams & DreamGen)!

Nvidia GEAR RSS 2025 Squad Rolling Out

🚀 GR00T Dreams code is live! NVIDIA GEAR Lab's open-source solution for robotics data via video world models. Fine-tune on any robot, generate 'dreams', extract actions with IDM, and train visuomotor policies with LeRobot datasets (GR00T N1.5, SmolVLA). github.com/NVIDIA/GR00T-D…

How we improve VLA generalization? 🤔 Last week we upgraded #NVIDIA GR00T N1.5 with minor VLM tweaks, FLARE, and richer data mixtures (DreamGen, etc.) ✨. N1.5 yields better language following — post-trained on unseen Unitree G1 with 1K trajectories, it follows commands on…

🚀 Introducing Cosmos-Predict2! Our most powerful open video foundation model for Physical AI. Cosmos-Predict2 significantly improves upon Predict1 in visual quality, prompt alignment, and motion dynamics—outperforming popular open-source video foundation models. It’s openly…

Assuming that we need ~2 trillion tokens to get to a robot GPT, how can we get there? I went through a few scenarios looking at how we can combine simulation data, human video data, and looking at the size of existing robot fleets. Some assumptions: - We probably need some real…

🔥 ReAgent-V Released! 🔥 A unified video framework with reflection and reward-driven optimization. ✨ Real-time self-correction. ✨ Triple-view reflection. ✨ Auto-selects high-reward samples for training.

Giving a talk about GR00T N1, GR00T N1.5, and GR00T Dreams in NVIDIA GTC Paris 06.11 2PM - 2:45PM CEST. If you are at Vivatech in Paris, please stop by the "An Introduction to Humanoid Robotics" Session!

Representation also matters for VLA models! Introducing FLARE: Robot Learning with Implicit World Modeling. With future latent alignment objective, FLARE significantly improves policy performance on multitask imitation learning & unlocks learning from egocentric human videos.

Nvidia also announced DreamGen, a new engine that scales robot learning with digital dreams It produces large volumes of photorealistic robot videos (using video models) paired with motor action labels and unlocks generalization to new environments

NVIDIA has published a paper on DREAMGEN – a powerful 4-step pipeline for generating synthetic data for humanoids that enables task and environment generalization. - Step 1: Fine-tune a video generation model using a small number of human teleoperation videos - Step 2: Prompt…

It’s not a matter of if, it’s a matter of when, video models and world models are going to be a central tool for building robot foundation models.

Getting robot data is difficult for those who don’t have the resources, and glad to see @NVIDIARobotics is offering an API for everyone to use!

Introducing 😶‍🌫️DreamGen, the pioneering approach to neural trajectories + robotics at NVIDIA GEAR lab. We’re among the first to show how large-scale synthetic data can significantly improve a robot’s ability to generalize to new actions and environments. If you’re interested,…

This has long been what was missing from video world models imo. Exciting progress.