Tom Davidson
@TomDavidsonX
Senior Research Fellow @forethought_org Understanding the intelligence explosion and how to prepare
New paper on AI-enabled coups. When AI gets smarter than humans, a few leaders could direct insane amounts of cognitive labor towards seizing power. In the extreme, an autonomous AI military could be made secretly (or not so secretly!) loyal to one person. What can be done? 🧵
Super interesting paper. If a misaligned AI generates a random string of numbers and another AI is fine-tuned on those numbers, the other AI becomes misaligned. But only if both AIs start from the same base model. This has consequences for preventing secret loyalties: - If an…
New paper & surprising result. LLMs transmit traits to other models via hidden signals in data. Datasets consisting only of 3-digit numbers can transmit a love for owls, or evil tendencies. 🧵
Really enjoyed doing this interview. We covered topics I haven't discussed before, like whether some coups would be worse than others and whether the US could gain global dominance by outgrowing the rest of the world
📻 New on the FLI Podcast! ➡️ @forethought_org's @TomDavidsonX and host @GusDocker discuss the growing threat of AI-enabled coups: how AI could empower small groups to overthrow governments and seize power. 🔗 Listen to the full interview at the link in the replies below:
METR's RCT finds that AI tools make developers *slower*! One reason: the RCT used "experienced codebase maintainers" who are 5-18x faster than the contractors used in METR's horizon length study. --> METR's horizon length estimates should be adjusted down. To automate AI RnD,…
This Twitter thread will contain various ad hoc thoughts on this bombshell @METR_Evals paper as they occur to me. Thinking live out loud! 🧵
My response to AI 2027: vitalik.eth.limo/general/2025/0… The AI 2027 post is high quality, I encourage people to read it at ai-2027.com I argue a misaligned AI will not be able to win nearly as easily as the AI 2027 scenario assumes, because it greatly underrates our…
Recommend this thread on a vision for compute-based international governance of AI, and concrete next steps
An International Atomic Energy Agency for AI? A Non-Proliferation Treaty for AI? CERN for AI? These ideas have been floating around for a while. But what could they actually look like? What could they actually do? What could incremental steps towards them be? My new paper is…
Should we give rights to future powerful agentic AIs? It might reduce the chance of harmful conflict with humans, and those AIs may inherently deserve certain rights. This question will become much more important as AI improves. Great to see a Forethought podcast on this!
New podcast episode with Peter Salib and Simon Goldstein on their article ‘AI Rights for Human Safety’. pnc.st/s/forecast/d76…
How concerned should we be about AIxBio? We surveyed 46 bio experts and 22 superforecasters: If LLMs do very well on a virology eval, human-caused epidemics could increase 2-5x. Most thought this was >5yrs away. In fact, the threshold was hit just *months* after the survey. 🧵
"They had really whipped themselves into a panic and seemed to believe that AI innovators were hell-bent on ushering in a complete technological takeover of humanity." Wait... But that's true actually... They really are trying to build superintelligence, which will automate…
Great evening at the @inferencemag debate on the potential for an intelligence explosion. Excellent contributions from @tylercowen, @TomDavidsonX, @NPCollapse and Mike Webb. Bravo @jackwiseman_ for organising and moderating!
Agree we should ground decisions about future AI rights in empirical measures. That doesn't mean AI will merit the same rights as humans. Their behavior and processing are v different to humans, and that may remain the case
I genuinely think "consciousness" is simply the modern, secular term for "soul". Both refer to unfalsifiable concepts used to determine who is in or out of our moral ingroup. Neither are empirical designations discovered through experiment, but socially constructed categories.
Interesting debate here about whether resource bottlenecks would block an industrial explosion
There’s just bottlenecks all over the place. For eg copper production which hasn’t even doubled in 25 years. At every step of the way there will have to be pauses to obtain new tech to do substitution or efficiencies. Sure it’s feasible, but it’s a very different political world.
This seems wrong. Today the benefits of AI outweigh the harms. I agree it's very likely future AI development is reckless, and we need to prepare to stop. But I'm not seeing plausible routes to global catastrophes today
Current AI development is reckless. We should expect it to remain so by default. There won't be some point where people suddenly stop recklessly racing, without serious effort to fix the underlying problems driving the current arms race. Or a lot of luck.
Nice to know the earth could support 100 trillion robots.
Seems a bit adversarial?
I like this analogy for AI chain-of-thought monitoring from @SydneyVonArx in the Financial Times
Interested in how AI-accelerated tech can/will work? Read this: lesswrong.com/posts/Na2CBmNY… (I think it's been ~10 years since LessWrong had a steady source of well-reasoned content like this. More please!)
I think the idea of the "industrial explosion" is of similar importance as the idea of the intelligence explosion, and much less discussed. Really happy to see this analysis.
To quickly transform the world, it's not enough for AI to become super smart (the "intelligence explosion") AI will also have to turbocharge the physical world (the "industrial explosion") New post lays out the stages of the industrial explosion, and argues it will be fast! 🧵