Nate Soares ⏹️

I wrote a book with Eliezer Yudkowsky. It’s about how smarter-than-human AI is on track to kill us all, written in hopes of bringing the conversation into the mainstream. Lots of people are alarmed; ~nobody wants to sound alarmist. The time is ripe. Preorders help. Link below.

Lots of people in AI, and especially AI policy, seem to think that aligning superintelligence is the most important issue of our time, and that failure could easily lead to extinction -- like what happened in AI 2027. But they don’t mention this fact in public because it sounds…

so many AI headlines over the past weeks bear out a simple point: AI companies can't reliably steer their models. 1. Anthropic can't guarantee that Claude 4 won't blackmail users if they make borderline requests x.com/shashj/status/…

I got to read a draft of this book (and I wrote a blurb!) and it's very good. The topic of AI alignment is complex and subtle, and this is the best unified summary of it I've read. Many of the online resources are scattered and piecemeal, and a lot of Eliezer's explanations…

Whether or not criminals deserve due process is beside the point. The point is that due process is how the state determines whether or not someone is a criminal.

No one has even tried this – a massive failure of civic engagement that should be rectified as soon as possible.

.@Aella_Girl, casually out of nowhere: "so I bought equipment to take a bath,"

My pet peeve is when AI people use the word "continuous" to mean something like "gradual" or "predictable" when talking about the future of AI. Y'all know this is a continuous function, right?

"We'll be fine (the pilot is having a heart attack but superman will catch us)" is very different from "We'll be fine (the plane is not crashing)". I worry that people saying the former are assuaging the concerns of passengers with pilot experience, who'd otherwise take the cabin

back when I was young, I thought it was unrealistic for the Volunteer Fire Department to schism into a branch that fought fires and a branch that started them

if EAs earnestly said "we're on track to die; yes some people have some plans but they're mostly predicated on the rest of the world drastically changing & that's looking less and less lkely; we're fucked", my guess is that'd be better than ~all their desperate plans combined

"our AIs that can't do long-term planning yet aren't making any long-term plans to subvert us! this must be becaues we're very good at alignment."

Emperor, Prince, Tsar, and Kaiser are all titles that stem from Roman desperation to call their ruler anything other than King

Americans: "I'll fucking do it, I'll cut my foot off" Canadians, revving chainsaw: "don't you dare. you think you're the only one who can cut their own foot off??"

This analysis of the path to AI ruin exhibits a rare sort of candor. The authors don't mince words or pull punches or act ashamed of having beliefs that most don't share. They don't handwring about how some experts disagree. They just lay out arguments. thecompendium.ai/#introduction

I’m proud to announce the April 1 launch of my new startup, Open Asteroid Impact! We redirect asteroids towards Earth for the benefit of humanity. Our mission is to have as high an impact as possible. 🚀☄️🌎💸💸💸 More details in🧵:

finally.