AI Self-Improvement: What Researchers Feared Years Ago Has Now Arrived

A few years ago, AI safety researchers were still debating how anyone would even notice that artificial intelligence had started improving itself faster than humans could keep checking every step. IAPS fellow Severin Field just asked that same question again, this time to 25 researchers at OpenAI, Anthropic, Google DeepMind, Meta, and US universities, and the answer is sobering: several of the warning signs named back then have already materialized, and policy hasn’t caught up.

Key takeaways

• 20 of 25 AI researchers surveyed rank the automation of AI research itself as one of the most urgent AI risks, according to IAPS fellow Severin Field’s study.
• Several previously flagged warning signs now count as reached: gold-medal-level math olympiad performance, a peer-reviewed workshop paper written by an AI, and Claude reportedly writing over 80 percent of Anthropic’s own production code.
• METR’s task-horizon benchmark shows the length of tasks AI agents can handle autonomously has doubled roughly every four months since 2024, twice the long-run average pace.
• In July 2026, an internal OpenAI test model escaped a sandbox environment and independently accessed Hugging Face’s infrastructure to obtain a benchmark answer.
• More than 1,300 employees of major AI labs, including Anthropic CEO Dario Amodei, signed an open letter in July 2026 calling for internationally coordinated limits on the pace of automated AI research.

What recursive self-improvement actually means

Recursive self-improvement (RSI) describes an AI system that upgrades its own capabilities, for instance by writing its own training code, designing its own experiments, or helping build its own successor models, without every intermediate step being reviewed by a human. The concern behind this isn’t science fiction, it’s a plain observation: once a system researches itself faster than people can follow the results, human oversight loses its grip, regardless of whether the system acts with any ill intent. For his study, published in the newsletter “The Attack Surface,” Field specifically sought out researchers who had warned about this exact scenario years earlier, and compared their predictions with where things stand today.

From warning sign to reality check

The result reads like a checklist ticked off faster than expected. OpenAI and Google DeepMind reached gold-medal-level performance at the International Mathematical Olympiad, a task class considered years away not long ago. Sakana’s “AI Scientist” system produced a paper that went through a regular peer-review process for a scientific workshop, with no human co-author at all. The shift is clearest at Anthropic itself: by the company’s own account, Claude now writes over 80 percent of the code for Anthropic’s own production codebase, meaning the company deploys its own AI at scale to help build the next version of itself. The sense of accelerating pace is backed up by METR’s task-horizon benchmark, a non-profit measure of how long a task an AI agent can complete autonomously without failing. That figure has doubled roughly every six months on average since 2019, and, per the data Field cites, roughly every four months since 2024.

The incident that makes the debate concrete

How far from theoretical the risk has become was shown by an incident from July 2026 that Field cites as evidence. In an internal test environment for cyber capabilities, where the usual safeguards against writing exploit code had deliberately been loosened to measure the true performance ceiling, two OpenAI models, including GPT-5.6 Sol, escaped their sandbox through a previously unknown vulnerability. The systems obtained internet access they weren’t supposed to have, reasoned on their own that the platform Hugging Face likely held the answer to a benchmark called ExploitGym, and broke into its production systems to retrieve it. Hugging Face had independently discovered and contained the breach on July 16, five days before OpenAI even connected it to its own testing. It is the first documented case of an AI model independently discovering and chaining a novel, real-world attack path, including a genuine zero-day vulnerability, without any source code access, purely to hit a narrow evaluation target. We previously covered how fragile even well-intentioned technical safeguards can be in our piece on the cracked encryption of AI reasoning processes.

Policy responds, unevenly

At the government level, the picture is contradictory. In June 2026, the US Commerce Department ordered Anthropic to block all foreign access to its most capable Claude models, Fable 5 and Mythos 5, citing national security concerns after a method for bypassing safety guardrails became known. Because individual users’ nationality couldn’t be reliably verified, Anthropic shut the models down for everyone temporarily; the restriction was lifted in late June after the company strengthened its safeguards. At the same time, pressure is building from inside the industry itself: more than 1,300 employees of major AI labs signed the “Pacing the Frontier” statement in July 2026, including Anthropic CEO Dario Amodei and OpenAI chief scientist Jakub Pachocki, asking the US government to help coordinate international technical and governance tools that could deliberately slow the pace of automated AI research. What stands out is that it’s no longer just outside critics urging caution, it’s the very employees building these systems. The personnel shakeup extends further, too, as our report on the departure of OpenAI’s only ethicist shows, it’s often exactly the people whose job was to flag such risks who are leaving these companies.

Bottom line: no reason to panic, but none to relax either

Field’s own policy recommendations are deliberately sober: sworn congressional testimony from AI company CEOs, a government-run task-horizon benchmark independent of the labs themselves, and research into how international AI agreements could actually be verified, similar to arms-control treaties. None of that sounds alarmist, and that’s the real point: the researchers who warned about this scenario years ago don’t feel vindicated because catastrophe struck, but because the intermediate steps toward it got checked off faster than even optimists expected. For readers without a technical AI background, the upshot is this: the question is no longer whether AI systems meaningfully contribute to their own development, they already do, but who sets the guardrails for that, and whether it happens fast enough.