Zhao-Heng Yin
@zhaohengyin
We introduce Dexterity Gen (DexGen), a foundation controller that enables unprecedented dexterous manipulation capabilities. For the first time, it allows human teleoperation of tasks such as using a pen, screwdriver, and syringe. Developed by @berkeley_AI and @MetaAI. A Thread.
Thrilled to share our recent advances in human teleoperation for dexterous manipulation, a level-up in robotic dexterity to previously unseen capabilities like using a syringe. This was work led by the amazing @zhaohengyin. Check out his thread for more details.
We introduce Dexterity Gen (DexGen), a foundation controller that enables unprecedented dexterous manipulation capabilities. For the first time, it allows human teleoperation of tasks such as using a pen, screwdriver, and syringe. Developed by @berkeley_AI and @MetaAI. A Thread.
Just open-sourced Geometric Retargeting (GeoRT) — the kinematic retargeting module behind DexterityGen. Includes tools for importing custom hands. Give it a try: github.com/facebookresear… A software by @berkeley_ai and @AIatMeta. More coming soon.
Artifacts in your attention maps? Forgot to train with registers? Use 𝙩𝙚𝙨𝙩-𝙩𝙞𝙢𝙚 𝙧𝙚𝙜𝙞𝙨𝙩𝙚𝙧𝙨! We find a sparse set of activations set artifact positions. We can shift them anywhere ("Shifted") — even outside the image into an untrained token. Clean maps, no retrain.
Very excited for this one. We took a cautiously experimental view on NN optimizers, aiming to find something that just works. SPlus matches Adam within ~44% of steps on a range of objectives. Please try it out in your setting, or read below for how it works.…
Everyday human data is robotics’ answer to internet-scale tokens. But how can robots learn to feel—just from videos?📹 Introducing FeelTheForce (FTF): force-sensitive manipulation policies learned from natural human interactions🖐️🤖 👉 feel-the-force-ftf.github.io 1/n
Excited to present FastTD3: a simple, fast, and capable off-policy RL algorithm for humanoid control -- with an open-source code to run your own humanoid RL experiments in no time! Thread below 🧵
The future of robotics isn't in the lab – it's in your hands. Can we teach robots to act in the real world without a single robot demonstration? Introducing EgoZero. Train real-world robot policies from human-first egocentric data. No robots. No teleop. Just Aria glasses and…
Robots need touch for human-like hands to reach the goal of general manipulation. However, approaches today don’t use tactile sensing or use specific architectures per tactile task. Can 1 model improve many tactile tasks? 🌟Introducing Sparsh-skin: tinyurl.com/y935wz5c 1/6
When @ToruO_O first shared this with me, I was quite impressed: "Wow, vision manipulation policy sim-to-real.” Incredible progress in exploring the potential of the pure sim-to-real approach!
Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids toruowo.github.io/recipe/ TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D
We consider teleoperation as a prompting problem, where the human action serves as a prompt to our foundation model, Dexterity Gen. Our model then generates the optimal action that ensures both stability and preservation of the human intent. Checkout our paper for more details.
We introduce Dexterity Gen (DexGen), a foundation controller that enables unprecedented dexterous manipulation capabilities. For the first time, it allows human teleoperation of tasks such as using a pen, screwdriver, and syringe. Developed by @berkeley_AI and @MetaAI. A Thread.
Teleop is all the rage for scaling, but can we even do dexterous tasks like tool-use with today’s systems? We built DexGen foundation controller, it takes coarse motion prompts to refine them to fine, safe actions. DexGen+teleop now enables tasks like screwdriver use & many more
We introduce Dexterity Gen (DexGen), a foundation controller that enables unprecedented dexterous manipulation capabilities. For the first time, it allows human teleoperation of tasks such as using a pen, screwdriver, and syringe. Developed by @berkeley_AI and @MetaAI. A Thread.
When @zhaohengyin first told me about this idea, I wasn't impressed. "How far can you get with teleoperation for hands?" I asked. Turns out, quite far :) really excited to see the unprecedented dexterity unleased by human motion prompt + sim-to-real, and look forward to more!!!
We introduce Dexterity Gen (DexGen), a foundation controller that enables unprecedented dexterous manipulation capabilities. For the first time, it allows human teleoperation of tasks such as using a pen, screwdriver, and syringe. Developed by @berkeley_AI and @MetaAI. A Thread.