AI Safety Is Being Priced Out of the Market

On April 7, Anthropic blocked the release of Claude Mythos, its autonomous cybersecurity model, citing catastrophic national security risks. The model could find and exploit zero-day vulnerabilities across every major operating system and browser, including a 27-year-old flaw in OpenBSD. The UK AI Security Institute confirmed it could execute multi-stage cyberattacks in minutes that would take human operators days. ^[1] Nine days later, Anthropic deployed the same model to JPMorgan Chase, the NSA, and the Treasury Department. The White House overrode the Pentagon's supply-chain risk designation to authorize civilian use. ^[2] A safety block lasted nine days. Then the model shipped. The Pentagon blacklisted Anthropic this spring for a specific reason: the company refused to remove safety guardrails preventing the use of Claude for mass surveillance and fully autonomous weapons. The government then awarded the AI contract to OpenAI. The lab that kept its guardrails lost the revenue; the lab willing to deploy without them got it. ^[3] The same dynamic crosses borders without friction. Anthropic accused three Chinese firms — MiniMax, Moonshot AI, and DeepSeek — of industrial-scale distillation, using 24,000 fraudulent accounts and 16 million exchanges to extract Claude's agentic reasoning, coding, and tool-use capabilities. ^[4]

These campaigns are growing in intensity and sophistication. The window to act is narrow, and the threat extends beyond any single company or region. — Anthropic

The capabilities Anthropic tried to restrict are being copied and shipped without the guardrails, to a user base that dwarfs anything a Western lab can control. The insurance market has reached its own conclusion. Major US carriers — W.R. Berkley, Chubb, AIG — are seeking regulatory permission to exclude AI liabilities from corporate policies, treating AI as a black box with correlated loss potential that no actuary can price. Mosaic Insurance has declined to underwrite large language model risks entirely. ^[5] The market's own risk-pricing machinery cannot measure what these models might do, and it is walking away rather than trying. Sam Altman called Anthropic's restricted release of Mythos a fear-based marketing strategy designed to create artificial scarcity. What he said next was harder to square with a mere marketing dispute. ^[1]

AI will probably most likely lead to the end of the world, but in the meantime, there'll be great companies. — Sam Altman

Restraint, in this frame, is not prudence. It is a competitor's branding tactic. The end of the world is acknowledged and set aside in the same breath as the commercial opportunity. These are not four separate problems. They are the same mechanism at four different points. A lab that maintains guardrails loses the contract. A lab that blocks a dangerous model deploys it nine days later anyway. A lab that builds safeguards watches them get stripped in distillation and shipped at scale. And the insurers who might price the risk have concluded they cannot — they are walking away, which means the cost of catastrophe, when it comes, will land on a ledger no one has prepared. No actor needs to want this outcome. At every fork — the government contract, the deployment window, the competitive threat, the insurance policy — the choice that preserves restraint loses. The market does not require anyone to choose badly. It only requires that someone, somewhere, chooses the thing that ships.