Yizhou Liu
@YizhouLiu0
PhD student at @MITMechE | Physics of living systems, Complex systems, Statistical physics
Superposition means that models represent more features than dimensions they have, which is true for LLMs since there are too many things to represent in language. We find that superposition leads to a power-law loss with width, leading to the observed neural scaling law. (1/n)
Elegant mapping! We should believe the existence something universal behind large complex systems — large language models included.
Interested in the science of language models but tired of neural scaling laws? Here's a new perspective: our new paper presents neural thermodynamic laws -- thermodynamic concepts and laws naturally emerge in language model training! AI is naturAl, not Artificial, after all.
Today we're releasing a developer preview of our next-gen benchmark, ARC-AGI-3. The goal of this preview, leading up to the full version launch in early 2026, is to collaborate with the community. We invite you to provide feedback to help us build the most robust and effective…
How much do language models memorize? "We formally separate memorization into two components: unintended memorization, the information a model contains about a specific dataset, and generalization, the information a model contains about the true data-generation process. When we…
Our interpretability team recently released research that traced the thoughts of a large language model. Now we’re open-sourcing the method. Researchers can generate “attribution graphs” like those in our study, and explore them interactively.
Are you a student or postdoc working on theory for biological problems? Just over two weeks left to apply for our fall workshop here at @HHMIJanelia. It’s free and we cover travel expenses! janelia.org/you-janelia/co…
Curious why disentangled representation is insufficient for compositional generalization?🧐 Our new ICML study reveals that standard decoders re-entangle factors in deep layers. By enforcing disentangled rendering in pixel space, we unlock efficient, robust OOD generalization!🙌
I just wrote a position paper on the relation between statistics and large language models: Do Large Language Models (Really) Need Statistical Foundations? arxiv.org/abs/2505.19145 Any comments are welcome. Thx
LLMs are Headless Chickens arxiv.org/abs/2505.20296
Your language model is wasting half of its layers to just refine probability distributions rather than doing interesting computations. In our paper, we found that the second half of the layers of the Llama 3 models have minimal effect on future computations. 1/6
🚀 Excited to share the most inspiring work I’ve been part of this year: "Learning to Reason without External Rewards" TL;DR: We show that LLMs can learn complex reasoning without access to ground-truth answers, simply by optimizing their own internal sense of confidence. 1/n
🎉 Excited to share our recent work: Scaling Reasoning, Losing Control 🧠 LLMs get better at math… but worse at following instructions 🤯 We introduce MathIF, a benchmark revealing a key trade-off: Key findings: 📉 Stronger reasoning ⇨ weaker obedience 🔁 More constraints ⇨…
It would be more impressive that LLMs can do what they can today without any of these capabilities. And even if they do not have now, it may not be hard to develop the abilities in future…
It is trivial to explain why a LLM can never ever be conscious or intelligent. Utterly trivial. It goes like this - LLMs have zero causal power. Zero agency. Zero internal monologue. Zero abstracting ability. Zero understanding of the world. They are tools for conscious beings.