Matthieu LC
@matthieulc
robots and llms / prev built typeless to make doctors happy, acq by @Doctolib
hi all, excited to join! i'm building an expressive mini "shoggoth" robot which will eventually be hooked up to gpt4o realtime voice. i'm currently working on the low-level policies, which are trained in a mujoco simulation with RL. to delay working on raw-pixels for now, i…
in dexterity tests, shoggoth mini reveals strong passive grasping generalization
wrapped up the blogpost and code for the shoggoth mini project! will release everything tomorrow 👀
i've been working on a blogpost about the project these past few days, hopefully released by the end of the week! here's a cute animation of the system design
just found out the white in human eyes is adaptive as the contrast makes gaze direction obvious, which helps in communicating intent much better than other animals sciencedirect.com/science/articl…
Learning where to look, learning what to see. Work on Continuous Thought Machines continues to be exciting! As a teaser, take a look at our approach to "Foveated Active Vision". Check it out 👇 (details in the thread) pub.sakana.ai/ctm
approaching the end of v1 here :) worked on some new primitives, and integrating them all together. making the gpt4o realtime api consistent in its use of the primitives was harder than i thought. i will now start writing the blogpost and cleaning the vibecode a bit for…
worked on having gpt4o drive open-loop motor primitives like waving or grabbing from stereo vision. special tokens like <distance: 10cm> get streamed straight into the transcript stream, which gpt4o reacts too. next up is integrating the closed-loop hand tracking. main challenge…
The Worldwide @LeRobotHF hackathon is in 2 weeks, and we have been cooking something for you… Introducing SmolVLA, a Vision-Language-Action model with light-weight architecture, pretrained on community datasets, with an asynchronous inference stack, to control robots🧵
sim2real achieved! mini shoggoth can now (slowly) track hands. to what purpose i do not yet know stack is PPO + mujoco + dynamics randomization + yolo11n
redesigned the mini shoggoth body to fit the stereo camera inside. also added a little spike at the tip to make it feel more alien. my roommate also made me notice that it looks a lot like a little green men head with a weird giger-like mandible when held upside down,…
finally got the rl tracking policy to work on the real robot this week! the tentacle loosely follows hands based on pose estimation of both hands and the tentacle tip. to do that i had to enhance/simplify the tip pose estimation + improve the hardware by covering the spool to…