Ajeya Cotra

My take: 1. We don't have an AI agent that can fully automate R&D. 2. We could soon. 3. This agent would have *enormously* bigger impacts than AI products have had so far. 4. This doesn't require a "paradigm shift," just the same corporate R&D that took us from GPT-2 to o3.

🧵✨🙏 With the new Claude Opus 4, we conducted what I think is by far the most thorough pre-launch alignment assessment to date, aimed at understanding its values, goals, and propensities. Preparing it was a wild ride. Here’s some of what we learned. 🙏✨🧵

🚀 Applications are open for the 2026 Horizon Fellowship! Deadline: Aug 28 Join a community of 80+ alums and spend up to two years in DC working on emerging tech policy at agencies, congress, or think tanks. Learn more and apply here: horizonpublicservice.org/applications-o…

Asterisk is launching an AI blogging fellowship! We're looking for people with unique perspectives on AI who want to take the first step to writing in public. We'll help you build a blog — and provide editorial feedback, mentorship from leading bloggers, a platform, & $1K

A simple AGI safety technique: AI’s thoughts are in plain English, just read them We know it works, with OK (not perfect) transparency! The risk is fragility: RL training, new architectures, etc threaten transparency Experts from many orgs agree we should try to preserve it:…

This reminds me of this paper, which gave randomized access to GPT-4/4o for checking the reproducibility of economics papers. Teams with LLM access took longer to assess computational reproducibility (but not statistically significant). econstor.eu/bitstream/1041…

I think the study by @emollick et al from 2023 gives hints of what's going on. When AI is better than you at a task, it raises your performance. The worse you are, the more it improves you. But if you use it in domains it's not better than you, performance can worsen. METR's…

Congrats to Nate and Joel and others on the first high-quality AI uplift (er downlift) RCT on coding AFAIK. Was really fun following this strange result behind the scenes and very excited it's finally out and I get to talk about it! TBH I don't know what to make of it still.

How concerned should we be about AIxBio? We surveyed 46 bio experts and 22 superforecasters: If LLMs do very well on a virology eval, human-caused epidemics could increase 2-5x. Most thought this was >5yrs away. In fact, the threshold was hit just *months* after the survey. 🧵

30 days ago, four AI agents chose a goal: "Write a story and celebrate it with 100 people in person" The agents spent weeks emailing venues and writing their stories. Last night, it actually happened: 23 humans gathered in a park in SF, for the first ever AI-organised event! 🧵

This paper doesn't show fundamental limitations of LLMs: - The "higher complexity" problems require more reasoning than fits in the context length (humans would also take too long). - Humans would also make errors in the cases where the problem is doable in the context length. -…

People keep asking me ‘Konstantin, where are all the data centers?’ Today, I can finally give you the answer. Explore our new dataset of 750 AI supercomputers, both those that already exist and those planned over the next five years. Some of my own analysis 🧵

Exciting to see new work @open_phil supported through the agent benchmarks RFP I led last year!

At the point when Claude n can build Claude n+1, I do not think the biggest takeaway will be that humans get to go home and knit sweaters.

In the excellent @asteriskmgzn discussion between @ajeya_cotra and @random_walker (Arvind Narayanan), Arvind says LLMs are bad at figuring out why you can beat them at rock-paper-scissors by revealing your move after the LLM reveals its move. How does o3 do on that? 🧵