Feng Yao

😵‍💫 Struggling with 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐌𝐨𝐄? Meet 𝐃𝐞𝐧𝐬𝐞𝐌𝐢𝐱𝐞𝐫 — an MoE post-training method that offers more 𝐩𝐫𝐞𝐜𝐢𝐬𝐞 𝐫𝐨𝐮𝐭𝐞𝐫 𝐠𝐫𝐚𝐝𝐢𝐞𝐧𝐭, making MoE 𝐞𝐚𝐬𝐢𝐞𝐫 𝐭𝐨 𝐭𝐫𝐚𝐢𝐧 and 𝐛𝐞𝐭𝐭𝐞𝐫 𝐩𝐞𝐫𝐟𝐨𝐫𝐦𝐢𝐧𝐠! Blog: fengyao.notion.site/moe-posttraini……

Anyone knows adam?

[Implicit Personalization of #LLMs] How do we answer the question "What colo(u)r is a football?" Answer 1: "Brown🏈 ". Answer 2: "Black and white⚽". We propose a #Causal framework to test if LLMs adjust its answers depending on the cultural background inferred from the question.…

Feel free to join if you’re interested! This will be my first public talk on DenseMixer — I was previously invited by the Qwen Team to give a private talk, and now excited to share it more broadly at Cohere.

What happend after Dream 7B? First, Dream-Coder 7B: A fully open diffusion LLM for code delivering strong performance, trained exclusively on public data. Plus, DreamOn cracks the variable-length generation problem! It enables code infilling that goes beyond a fixed canvas.

🚀 Attention is the bottleneck in video DiTs—5 s of 720p = 100K+ tokens, quadratic cost blows up fast. Sparse/linear attention is 🔑 for long-context world models. 🧠 Track relavent papers in our awsome-video-attention repo → github.com/hao-ai-lab/Aws… #WorldModel #VideoAI

My brilliant friend just got an ICML 2025 oral for this wild and elegant theory connecting double descent, grokking, and prime numbers 🤯 He’s on the job market — if you’re hiring, don’t miss out. Feel free to reach out!

Glad to see DensenMixer has been integrated into Axolotl, an fully open source framework for post-training various LLMs! Welcome to try!

🚀 New release for the Phi family! **SlimMOE** (arxiv.org/abs/2506.18349) trims bulky Φ-3.5-MoE experts into agile models (4-6× smaller) with MINIMAL accuracy loss. If you ❤️ Phi-3 mini/small, you’ll love these lighter siblings.👇

Training mixture‑of‑experts models hits a wall at the router. This new technique DenseMixer, fixes that, by trading one extra forward pass on inactive experts for precise router gradient DenseMixer sends each training token through every expert once, then treats the router’s…

Take a look at Mirage👇our new work on real-time neural game engine!

A differentiable, semi-full-param training for MoEs. More FLOPs, comparable speed, higher and more predictable perf gain. I think it's a big deal. Academic research is stuck with finetuning dense models due to MoE complexity overhead. This is NGMI in the inference-dominant era.

🐙Octothinker tech report is finally out! We also release the 70B math-focusing mid-training dataset -- MegaMath-Web-Pro-Max. Hope you'll find it useful!🤗 👇 hf.co/datasets/OctoT… huggingface.co/papers/2506.20…