Junyao Shi
@JunyaoShi
🤖PhD @Penn @GRASPlab, Incoming Research Intern @SkildAI, Previously @Columbia | Robot Learning
💡Can robots autonomously design their own tools and figure out how to use them? We present VLMgineer 🛠️, a framework that leverages Vision Language Models with Evolutionary Search to automatically generate and refine physical tool designs alongside corresponding robot action…
Very cool new dexterous robot hand
We at @DaxoRobotics found a new (and better) way to build towards true robot dexterity. This dexterous robotic hand is something I’ve been working on since graduating from @GRASPlab. Below is just a teaser. Enjoy the spin. The full story drops tomorrow.
We at @DaxoRobotics found a new (and better) way to build towards true robot dexterity. This dexterous robotic hand is something I’ve been working on since graduating from @GRASPlab. Below is just a teaser. Enjoy the spin. The full story drops tomorrow.
How to generate billion-scale manipulation demonstrations easily? Let us leverage generative models! 🤖✨ We introduce Dex1B, a framework that generates 1 BILLION diverse dexterous hand demonstrations for both grasping 🖐️and articulation 💻 tasks using a simple C-VAE model.
I wrote a fun little article about all the ways to dodge the need for real-world robot data. I think it has a cute title. sergeylevine.substack.com/p/sporks-of-agi
Official results are in - Gemini achieved gold-medal level in the International Mathematical Olympiad! 🏆 An advanced version was able to solve 5 out of 6 problems. Incredible progress - huge congrats to @lmthang and the team! deepmind.google/discover/blog/…
At a robotics lab in Pittsburgh, engineers are building adaptable, AI-powered robots that could one day work where it's too dangerous for humans. The research drew a visit from President Trump, who touted U.S. dominance in AI as companies announced $90 billion in new investments.
By nearly every metric, Muscle v0 is the most capable dexterous robotic hand I’ve ever seen. Most popular hands today are little more than clumsy grippers—longstanding bottlenecks for progress in robot learning. Muscle v0 changes that. Here’s what sets it apart: •HUNDREDS of…
Introducing Muscle v0 -- infinite degrees of freedom, from @DaxoRobotics. A different mountain to climb - with a far more beautiful peak. We built this from the ground up: - Ultra-dexterous - Built for machine learning - Durable and robust More below (1/n)
3D print tactile sensors anywhere inside your fin-ray fingers! We present FORTE - a solution to sensorize compliant fingers from inside with high resolution force and slip sensing. 🌐merge-lab.github.io/FORTE/ With precise and responsive tactile feedback, FORTE can gently handle…
Incredible! I think CoRL, RSS, ICRA etc. should have a separate track for companies to showcase their VLA demos in-person. This gives people much better understanding than looking at well-edited videos on their websites.
We have started taking DYNA-1, our dexterous robust VLA model, to conferences and showcasing it for hours on end! The model run for 3 days, 8 hours each day at #HITEC2025 3 weeks ago with 99.9% overall success rate (dropped 1 towel in day 2). No intervention, it just works :)
It's so satisfying to watch our models just do the task on demand; human like and no downtime
We have started taking DYNA-1, our dexterous robust VLA model, to conferences and showcasing it for hours on end! The model run for 3 days, 8 hours each day at #HITEC2025 3 weeks ago with 99.9% overall success rate (dropped 1 towel in day 2). No intervention, it just works :)
Current robot policies often face a tradeoff: they're either precise (but brittle) or generalizable (but imprecise). We present ViTaL, a framework that lets robots generalize precise, contact-rich manipulation skills across unseen environments with millimeter-level precision. 🧵
Well said! In our recent ICRA 2025 paper, ZeroMimic (zeromimic.github.io), we demonstrate how robots can directly learn manipulation skills from large-scale human web videos. Leveraging diverse, abundant data sources is especially critical in robotics, where collecting data…
this is a great read by jack morris. gives such a fresh perspective on what really matters. TLDR: with every "new" architecture, we unlocked a new source of data to use at scale. it was the large amount of data we unlocked that boosted performance not the architecture itself,…
🤖 Household robots are becoming physically viable. But interacting with people in the home requires handling unseen, unconstrained, dynamic preferences, not just a complex physical domain. We introduce ROSETTA: a method to generate reward for such preferences cheaply. 🧵⬇️
Thanks for your well-put summary of VLMgineer @IlirAliu_ ! We are indeed excited about applying VLMgineer to more advanced hardware design automation in the future. Stay tuned!
🛠️ What if a robot could invent its own tools. And teach itself how to use them? That’s exactly what VLMgineer does: a new framework that lets Vision Language Models (VLMs) design physical tools and the actions to use them, entirely on their own. No templates. No human…
We’re bringing powerful AI directly onto robots with Gemini Robotics On-Device. 🤖 It’s our first vision-language-action model to help make robots faster, highly efficient, and adaptable to new tasks and environments - without needing a constant internet connection. 🧵
Very happy that EgoDex received Best Paper Awards of 1st EgoAct workshop at #RSS2025! Huge thanks to the organizing committee @SnehalJauhri @GeorgiaChal @GalassoFab10 @danfei_xu @YuXiang_IRVL for putting out this forward-looking workshop. Also kudos to my colleagues @ryan_hoque…
In RoboTool (arxiv.org/abs/2310.13065), we showed that LLM provides robots with important prior knowledge about how to use tools. This time, @JunyaoShi and his colleagues showed that VLM not only provides guidance about how to use tools, but also how to design them. Check out this…
💡Can robots autonomously design their own tools and figure out how to use them? We present VLMgineer 🛠️, a framework that leverages Vision Language Models with Evolutionary Search to automatically generate and refine physical tool designs alongside corresponding robot action…