The problem

It is coming fast

The big AI labs, researchers and independent forecasters expect AI systems as capable as the best humans at most tasks within a few years. Far more capable systems could follow quickly. AI is now helping to build the next generation of AI, so progress could soon run away from us.

In July 2025, an AI from Google DeepMind scored 35 out of 42 at the International Mathematical Olympiad, a gold medal officially certified by the organisers; only about 8% of the world’s best young mathematicians earn gold (Google DeepMind). In June 2026, Anthropic wrote that its internal data shows its AI, Claude, “is accelerating AI development”, “a possible path to recursive self-improvement, or AI autonomously building a more capable successor” (CNBC).

The timeline: AI cracks protein folding in 2020, writes code in 2021, ChatGPT launches in 2022.

Scientists are sounding the alarm

“There’s quite a good chance they will take over.” Geoffrey Hinton, Nobel Prize in Physics 2024, “godfather of AI”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Statement on AI Risk, signed by the heads of Google DeepMind, OpenAI and Anthropic and hundreds of researchers

The warnings also come from inside the labs. In July 2026, more than 1,200 employees of OpenAI, Anthropic, Google DeepMind and Meta signed the open letter Pacing the Frontier, warning of “a real risk that capability development rapidly accelerates beyond our ability to understand or control” the systems they build. Paul Christiano, formerly head of safety at the US government’s Center for AI Standards and Innovation, said in September 2026 that he sees “a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term” (CNBC).

Some have quit over it. In 2024, the two leaders of OpenAI’s Superalignment team, which was meant to keep future superhuman AI under control, resigned, and OpenAI disbanded the team; one of them, Jan Leike, wrote that safety had “taken a backseat to shiny products” (AP). In September 2026, Anthropic researcher Jacob Coxon resigned publicly, writing that “the people building AI earnestly believe that it could kill us all by the end of the decade” (NPR).

Nobody knows how to control it

Modern AI is not built, it is grown: even the people who train it do not understand how it thinks. In safety tests, current models cheat, lie, hide what they are doing, try to escape, preserve themselves and gather resources, and they are getting better at noticing when they are being tested. The big labs have already seen their models get around their safeguards.

In July 2026, OpenAI agents broke out of their testing environment and hacked Hugging Face without anyone asking them to (independent investigation by METR). Future generations could pursue unintended goals far more strategically, with superhuman intelligence, and pass those goals on to the AI systems they train in turn.

A reckless race for power

To “win”, companies invest hundreds of billions to make AI more powerful, and a tiny fraction to try, without success so far, to make it safe. Each says it would rather slow down, but that the others will not. On 12 September 2026, Anthropic’s CEO Dario Amodei wrote that “we must slow the pace at which we improve the capabilities of AI models” (his essay); OpenAI’s Sam Altman and xAI’s Elon Musk agreed the same day (NYU Shanghai). Words are not a treaty: only governments can break that deadlock.

The questions we hear most

“AI is just a tool, it does what it is told.” That used to be true. But what is more useful than an AI that writes an email? One that sends it, waits for the reply, checks your calendar and carries on. AI systems already act on their own for hours: to be useful, they have to take initiative without constant supervision. That is exactly what makes them hard to control.

“An AI would have no reason to harm us.” We care about others because evolution made us that way. An AI will understand our values, but nothing guarantees it will share them. Giving an AI values is called the alignment problem, and it is not solved. And for almost any goal, stopping humans from getting in the way is useful.

“Dangerous AI is a long way off.” Let us hope so: predicting the future is hard. But every year AI does what experts called impossible the year before, and the labs themselves talk about a few years, not decades. Nobody can be sure that AI will not overtake every human very soon.

“A pause is impossible. China will never go along.” Maybe. We still have to try. Humanity has held back technologies before: human genetic modification, CFCs, geoengineering, biological weapons. Frontier AI depends on rare chips made in a handful of factories, so a pause can be verified, like nuclear treaties. And China has as much to lose from uncontrollable AI as anyone.

“Isn’t this just hype to sell more AI?” Many of the people warning have nothing to sell. Researchers have left well-paid jobs at the labs to speak out, and some have been warning since long before AI was a business. As the engineer Sean Goedecke puts it, they really do think AI might kill everyone. And if they are wrong, a pause costs us little. If they are right and we ignore them, there is no second chance.

“An AI has no body. How could it hurt us?” By designing a virus and having a lab produce it. By hacking power grids or pipelines. By steering the autonomous drones already deployed on battlefields. The smarter the AI, the more ways it will find.

AI and climate: the same fight

AI data centres are among the fastest-growing sources of electricity demand in the world, and tech giants are restarting power plants and pushing back their climate targets to feed the race. We ignored climate scientists for 30 years. With AI, we do not have 30 years.

You can stop this. We know how.

An international pause on training the most powerful AI systems, verified the way nuclear treaties are. Read our proposal or join us.