Rune Kvist
@RuneKvist
Build the incentive systems you want to see in the world Prev: first product hire @AnthropicAI, board @ Center for AI Safety
Insurance is an underrated way to unlock secure AI progress. Insurers are incentivized to truthfully quantify and track risks: if they overstate risks, they get outcompeted; if they understate risks, their payouts bankrupt them. 1/9
Insurance markets have a significant role to play in driving the 'flywheel' of AI governance at a rate closer to the rate at which the tech is developing. An important effort alongside regulatory markets, which we discussed here (and @RuneKvist cites) srinstitute.utoronto.ca/news/co-design…
🧐
.@Replit goes rogue during a code freeze and shutdown and deletes our entire database
"went under the radar" clearly need to learn to poast better
went under the radar but here’s yet another AGI manifesto that’s really worth reading
To summarize this week: - we released general purpose computer using agent - got beaten by a single human in atcoder heuristics competition - solved 5/6 new IMO problems with natural language proofs All of those are based on the same single reinforcement learning system
AI progress in math is way ahead of even the most optimistic predictions Most importantly, it’s a general reasoning model rather than a narrow math-specific model. These reasoning abilities will transfer to other domains
Today, we at @OpenAI achieved a milestone that many considered years away: gold medal-level performance on the 2025 IMO with a general reasoning LLM—under the same time limits as humans, without tools. As remarkable as that sounds, it’s even more significant than the headline 🧵
To put it mildly, the past week at Windsurf has been crazy. There have been a lot of different rumors and reports, so I want to share a transparent account of how it actually went down. Before I start, I just want to say that Varun and Douglas were great founders and this…
I wouldn't want to mandate an insurance model prematurely, but when the product in question contains disclaimers like "use at your own risk, may leak your company data and drain your bank account," OpenAI may want to take out a liability insurance policy on their own accord.
Welp, here we go. I hope we can build some good ways to scope and steer these agents reliably. I think we should require that @OpenAI sell an insurance product for their agents. I think a market tied to the level of claimed damages and adjudicated damages would be great.
I'm deeply inspired by this articulation of the philosopher-builder.
Every builder's first duty is philosophical: to decide what they should build for. AI is beginning to decide what ideas reach your mind—your next action, your next job, your next relationship. It will tempt you to outsource your thinking in ways you’ve never been tempted before.…
Ben Franklin as philosopher-builder:
I like the archetype of Ben Franklin as the philosopher-builder:
I agree. Now, we'll also need a new actuarial evolution. The current paradigm focuses on historical data. AI changes fast, lots of risk is brand new. Historical → Real-time Base rates → Specifics Top down → Bottom up
the actuarial revolution was a precondition of the industrial revolution, it will play a role again
croesus made a shitload of money with a market solution to urban fires, and bankrolled julius caesar’s political campaigns. by some measures the richest man in history. just saying.
I can imagine how fire prevention would’ve seemed impossible to solve with market solutions in the first large waves of urbanization. Finding situations where markets can help price the risk of catastrophe, or even just small incidents, and incentivize its mitigation seems great.
at first it seems impossible to insure against asi xrisk because in the event of total extinction nobody claims their policy. but you could make it work the other way around; i can take out a policy against your x-risk claim.
"AI insurance is capitalism’s solution to AI safety risk."
AI insurance is capitalism’s solution to AI safety risk. Put money behind the claims and incentivize a system to prevent runaway problems.
Insurance: the Ultimate AI Eval?
Insurance is an underrated way to unlock secure AI progress. Insurers are incentivized to truthfully quantify and track risks: if they overstate risks, they get outcompeted; if they understate risks, their payouts bankrupt them. 1/9
Love a lot of this, great foundations! I’d add one to your list of 25 though. Re/insurers don’t currently have a clear view of the AI-related risks in their portfolios. Work needs to be done to understand how AI is already embedded in their portfolios and how that’s increasing at…
I can imagine how fire prevention would’ve seemed impossible to solve with market solutions in the first large waves of urbanization. Finding situations where markets can help price the risk of catastrophe, or even just small incidents, and incentivize its mitigation seems great.
Insurance is an underrated way to unlock secure AI progress. Insurers are incentivized to truthfully quantify and track risks: if they overstate risks, they get outcompeted; if they understate risks, their payouts bankrupt them. 1/9
The most relevant AI risk eval:
Still waiting for this leaderboard
Yes! We got lots of great insights on this during our regulatory markets workshop @TorontoSRI in 2024. Insurance industry has nose for connecting capital to quantified risk. Motivates risk quantification industry into existence.
Excited to see this in the world. Even the fastest racecars need seatbelts 🏎️