Toby Ord

The Precipice is finally out today in the US and Canada. And in audiobook form everywhere. Though this is a dark time, the book is ultimately hopeful—pointing the way to an extremely bright future that our actions can secure.

How much past is behind us and how much future might be left ahead? I made an interactive timeline visualizing how answers to this question have changed over the generations. It was published in the @antikythera_xyz journal, in collaboration with @mitpress. Link below!

Today we're releasing Community Alignment - the largest open-source dataset of human preferences for LLMs, containing ~200k comparisons from >3000 annotators in 5 countries / languages! There was a lot of research that went into this... 🧵

METR previously estimated that the time horizon of AI agents on software tasks is doubling every 7 months. We have now analyzed 9 other benchmarks for scientific reasoning, math, robotics, computer use, and self-driving; we observe generally similar rates of improvement.

Reward Learning is just supervised learning, and so should be equally safe, right? Wrong! Our paper “The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret” shows that policy optimization causes issues. It was accepted to ICML! 🧵

We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers. The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.

The first-ever UN Global Risk Report has been released. Over 1,100 experts offer their insights into the risks most likely to shape our future and assess how prepared the world is to address them. Full report: unglobalriskreport.org

STAGGERING: This new study of 133 countries is the first to estimate the impact of all USAID’s work. In 2 decades, it saved *92M* lives. Current cuts, if not reversed, are forecast to cost *14M* lives thru 2030. thelancet.com/journals/lance…

📢 £18m grant opportunity in Safeguarded AI: we're looking to catalyse the creation of a new UK-based non-profit to lead groundbreaking machine learning research for provably safe AI. Learn more and apply by 1 October 2025: link.aria.org.uk/ta2-phase2-x

Enjoyed the recent @80000Hours w/ @tobyordoxford. Agree that AI policy researchers should dream bigger on societal Qs. @simondgoldstein and I have been working on one of Toby's big questions: Should the AGI economy be run like a slave society (as it will under default law)?

A surprising & little-known results in classical statistics: Mean (μ) and median (m) are within one std deviation: |μ−m| ≤ σ For unimodal densities, bound is even tighter |μ−m| ≤ 0.7746 σ This beautiful results first appeared in a 1932 paper by Hotelling & Solomons 1/3