Russell Mendonca
@mendonca_rl
Optimus AI @Tesla. Prev - PhD student @CMU_Robotics
A single neural net has learned all these interaction skills, primarily from video. It has been very exciting to push on this effort for generalist models over the past few months as part of the team (shoutout to @Yi__Li and @ashishkr9311 ). We’re going to see continued…
I’m not just dancing all day, ok
BREAKING: First ever Tesla Model Y robotaxi with no-one in the drivers seat spotted testing on public roads in Austin, Texas! Tesla's new "Robotaxi" wordmark/logo is on the side of the vehicle.
HOLY CRAP ITS A #ROBOTAXI!! @SawyerMerritt @WholeMarsBlog @DirtyTesLa @niccruzpatane
I forgot to mention that Optimus was doing this while “blindfolded”. We haven’t yet made vision part of the control loop for uneven terrain. Optimus robots are regularly cruising around our workplace in Palo Alto where vision is used for static & dynamic object avoidance.
Tesla is where real-world AI is happening. These runs are on mulched ground, where I’ve myself slipped before. What’s really crazy here is that for these, Optimus is actually blind! Keeping its balance without video (yet), only other on-board sensors consumed by a neural net…
Tesla is where real-world AI is happening. These runs are on mulched ground, where I’ve myself slipped before. What’s really crazy here is that for these, Optimus is actually blind! Keeping its balance without video (yet), only other on-board sensors consumed by a neural net…
Daily walks help clear your mind
Super congratulations to the @Tesla_AI software & chip design teams on a successful @Robotaxi launch!! Culmination of a decade of hard work. Both the AI chip and software teams were built from scratch within Tesla.
Interesting work on learning more efficiently from demos. Learns a world models that allows search at inference time.
Say ahoy to 𝚂𝙰𝙸𝙻𝙾𝚁⛵: a new paradigm of *learning to search* from demonstrations, enabling test-time reasoning about how to recover from mistakes w/o any additional human feedback! 𝚂𝙰𝙸𝙻𝙾𝚁 ⛵ out-performs Diffusion Policies trained via behavioral cloning on 5-10x data!
For the past several days, Tesla has been testing self-driving Model Y cars (no one in driver’s seat) on Austin public streets with no incidents. A month ahead of schedule. Next month, first self-delivery from factory to customer.
The exponential increase in data will lead to a step change for general AI for applications beyond robotics. Pre-training datasets will not be limited to the internet.
Imagine you had a humanoid robot. You could teach it to do anything. When it recharged, it could upload its learnings (on an anonymized basis) to all the other robots. It could be the most powerful data network effect in history.
One of our goals is to have Optimus learn straight from internet videos of humans doing tasks. Those are often 3rd person views captured by random cameras etc. We recently had a significant breakthrough along that journey, and can now transfer a big chunk of the learning…
I’m not just dancing all day, ok
The sim2real RL pipeline we’ve built scales to many different challenging motions. This tech will soon be extended to useful manipulation tasks.
Was just getting warmed up
Early steps in building a legion for the next level of data scaling !
Me, Robot
Really impressive generalization results, more evidence that data scaling with powerful ML models will work for robotics.
π-0.5 is here, and it can generalize to new homes! Some fun experiments with my colleagues at @physical_int, introducing π-0.5 (“pi oh five”). Our new VLA can put dishes in the sink, clean up spills and do all this in homes that it was not trained in🧵👇
We're continuing to leverage RL for Optimus, coming soon for manipulation tasks as well! Come join us to build towards generalist robots, utilizing simulation and videos.
Accurate actuators accelerate automation
1.5 yrs ago, we set out to answer a seemingly simple question: what are we *actually* getting out of RL in fine-tuning? I'm thrilled to share a pearl we found on the deepest dive of my PhD: the value of RL in RLHF seems to come from *generation-verification gaps*. Get ready to🤿!
Was really rewarding pushing on getting Optimus to confidently walk outside the lab on challenging terrain, in my first couple of months! I am blown away by the drive, ambition, vision and resources of the team @Tesla_Optimus. Come join us to make robot learning sci-fi dreams a…
Daily walks help clear your mind
These hands balance high controllability, reactivity and power, without blowing up design size. Very exciting ! In personal news, I recently completed my PhD from CMU, and have joined Optimus AI ! Would like to thank @pathak2206 for his guidance and mentorship.
Got a new hand for Black Friday
We found using an exoskeleton based approach to be much more effective than VR for teleop. Likely due to higher accuracy and lower latency since this doesn’t rely on vision estimation of hand positions, instead directly commanding robot joint positions.
Teaching bimanual robot hands to perform very complex tasks has been notoriously challenging. In our work, Bidex: Bimanual Dexterity for Complex Tasks, we’ve developed a low-cost system that completes a wide range of highly dexterous tasks in real-time. bidex-teleop.github.io
Dense language annotations will enable us to do more with a given dataset in robotics, just as in image/video settings. The effectiveness of this approach will only improve as foundation models get better at producing detailed language labels given behavior clips.
Excited to share our work on STEERing robot behavior! With structured language annotation of offline data, STEER exposes fundamental manipulation skills that can be modulated and combined to enable zero-shot adaptation to new situations and tasks.