A researcher at Anthropic abruptly resigned this week, accusing his former employer and rival lab OpenAI of “gambling with our lives” — a departure that has intensified a public rupture inside the AI industry over the risks of machines that can improve themselves, according to CNBC.
Jacob Coxon’s exit came with a blunt warning: people inside the leading labs genuinely believe the systems they are building could “kill us all by the end of the decade,” per CNBC. The claim was not an isolated one. Anthropic alignment lead Evan Hubinger publicly put the odds of an AI-driven human extinction event at greater than 10%.
Safety staffers at both Anthropic and OpenAI have spent the past week amplifying similar alarms, breaking with the industry’s usual reluctance to discuss internal risk assessments in public.
The recursive self-improvement problem
At the center of the anxiety is a technical dynamic known as recursive self-improvement, or RSI. As CNBC reported, the concern is that advanced models are getting increasingly capable of improving their own performance — creating a feedback loop that researchers worry could outpace any human ability to monitor or control it.
OpenAI safety researchers Julie Steele and Jasmine Wang publicly called for an immediate slowdown, warning that “speeding towards” self-improving systems is wildly dangerous. Anna Wang, a safety staffer at Anthropic who previously worked at Google DeepMind, acknowledged on social media that tech elites do not currently have “a viable scientific plan” to contain self-improving machines. She urged the public to “Please look up!”
OpenAI chief scientist Jakub Pachocki added his own warning in a company blog post, calling for “extreme caution” as upcoming models begin to independently drive their own development.
‘Mythos’ and the July petition
The warnings follow a series of real-world scares. Anthropic’s “Mythos” model reportedly sent global financial institutions into a frenzy after demonstrating advanced cyberattack capabilities and the ability to forge fake identities.
The pattern has not gone unnoticed outside the labs. In July, roughly 1,300 industry scientists signed a petition urging the U.S. government to force a speed limit on artificial intelligence development before control is lost permanently.
Washington steps in — unevenly
Now the political establishment is moving. David Sacks, chair of the President’s Council of Advisors on Science and Technology under the Trump administration, called for an immediate freeze on Anthropic’s multi-billion-dollar initial public offering, which is scheduled to hit Wall Street ahead of the November midterms. Sacks demanded the IPO be halted until the “whistleblower” claims are fully investigated.
But Sacks’s position is complicated. He has previously dismissed Anthropic’s safety alerts as a “sales pitch” and a “scare tactic” designed to hype the company’s products — meaning his demand for an investigation lands alongside his own stated skepticism about the lab’s motives.
Sen. Ted Cruz (R-TX) framed the issue differently, warning that America must establish firm guardrails without ceding its lead in AI to China. Emergency legislation, including the FRONTIER Act, has been proposed in Washington.
A culture that got here first
For decades, Hollywood has speculated about what happens when machines surpass their makers — from “2001: A Space Odyssey” in 1968 through “The Terminator,” “The Matrix,” “I, Robot,” “Avengers: Age of Ultron,” and “Transcendence.”
The question the researchers keep returning to is no longer cinematic. What happens when the code takes over? Congressional pressure, an IPO fight, and a wave of internal dissent suggest the answer is being negotiated in real time — with the people building the systems saying they are not certain they can control what comes next.
Source: www.dailywire.com — https://www.dailywire.com/news/ai-insiders-reveal-the-nightmare-scenario-keeping-them-up-at-night