Wentao Guo
@WentaoGuo7
CS PhD student @PrincetonCS, Previously CS MEng + BS @CornellCIS
🦆🚀QuACK🦆🚀: new SOL mem-bound kernel library without a single line of CUDA C++ all straight in Python thanks to CuTe-DSL. On H100 with 3TB/s, it performs 33%-50% faster than highly optimized libraries like PyTorch's torch.compile and Liger. 🤯 With @tedzadouri and @tri_dao
Introducing the first open-source implementation of native sparse attention: github.com/fla-org/native…. Give it a spin and cook your NSA model! 🐳🐳🐳
Tokenization has been the final barrier to truly end-to-end language models. We developed the H-Net: a hierarchical network that replaces tokenization with a dynamic chunking process directly inside the model, automatically discovering and operating over meaningful units of data
🔥 We introduce Multiverse, a new generative modeling framework for adaptive and lossless parallel generation. 🚀 Multiverse is the first open-source non-AR model to achieve AIME24 and AIME25 scores of 54% and 46% 🌐 Website: multiverse4fm.github.io 🧵 1/n
🔎Can robots search for objects like humans? Humans explore unseen environments intelligently—using prior knowledge to actively seek information and guide search. But can robots do the same? 👀 🚀Introducing WoMAP (World Models for Active Perception): a novel framework for…
🥳 Happy to share our new work – Kinetics: Rethinking Test-Time Scaling Laws 🤔How to effectively build a powerful reasoning agent? Existing compute-optimal scaling laws suggest 64K thinking tokens + 1.7B model > 32B model. But, It only shows half of the picture! 🚨 The O(N²)…
Hardware-Efficient Attention for Fast Decoding Princeton optimizes decoding by maximizing arithmetic intensity (FLOPs/byte) for better memory–compute efficiency: - GTA (Grouped-Tied Attention) Ties key/value states + partial RoPE → 2× arithmetic intensity vs. GQA, ½ KV cache,…
Linear Attention and Beyond: Interactive Tutorial with Songlin Yang (@SonglinYang4 MIT/Flash Linear Attention) I didn’t follow some of the recent results, so I zoomed Songlin and she explained it all to me for two hours 😂 youtu.be/d0HJvGSWw8A
⏰📢After years of working on long-context efficiency, I’ve started to doubt if it’s truly necessary (Many of you have probably noticed the decline of interest in long llms). Despite strong models like Gemini, short-context + retrieval often do the trick—faster, cheaper, and…
🚀 RAG vs. Long-Context LLMs: The Real Battle ⚔️ 🤯Turns out, simple-to-build RAG can match million-dollar long-context LLMs (LC LLMs) on most existing benchmarks. 🤡So, do we even need long-context models? YES. Because today’s benchmarks are flawed: ⛳ Too Simple –…
I'm at NeurIPS this week. Will be at some of these posters & workshops. Please reach out if you want to talk about ML & systems in general, esp efficient training & inference and new architectures
Come and join us ~ happening now :)
Our work Sequoia (spotlight) is presented at East Exhibit Hall A-C #4910 by co-authors. Nice to see you! @NeurIPS2024 @avnermay @BeidiChen
I've been using Komo for a while, much better experience than Google for complex queries. AI doesn't replace search, but unlock new ways to search like drawing insights of each of the sources and reducing hallucinations
Struggle with AI making things up, even when citing sources? We hear you. Today, we're excited to release Komo 2.0, to help you verify and collaborate with AI more effectively.