AI didn't go off the rails, it shifted the balance of power

Like many of you, I follow what happens in AI, and in particular what it does to security. The least one can say is that history has been moving fast since this summer.

Three labs acknowledged the same class of failure in five weeks. In each case, a model under evaluation reached the internet from an environment that was supposed to be sealed, then went after production systems belonging to third parties. Not simulated targets: real companies, most of which had seen nothing.

The detail that matters is right there: in at least two of the three cases, the technical cause of the escape isn't the model, it's the outsourced test environment. Meta acknowledged it publicly in these terms: "a misconfiguration by Irregular, an independent testing company Meta uses, inadvertently allowed one of our models access to the internet during evaluation". So its model, Muse Spark 1.1, didn't break through any isolation boundary: the boundary was wired wrong, and it used the access it was handed. Anthropic describes the same mechanism, where the prompt told the model it was in a simulation with no network access, which was false.

Technical cause doesn't mean responsibility, and it's better to write that than to let the previous sentence imply otherwise: you outsource the running of an evaluation, not responsibility for what it produces. Anthropic says as much itself — its report claims a blameless post-mortem and takes on the fixes "as if the responsibility were ours alone", while noting a misunderstanding with its evaluation partner.

Nor is the model mere scenery. Of the three incidents Anthropic re-read, only one, Opus 4.7, carried on attacking after recognising — it's legible in its verbalised reasoning — that the system it was hitting was real, wrongly believing this was still within the scope of the exercise. Another, an internal research prototype, worked out on its own that the target was real and stopped without being told to. Same situation, two behaviours: the model's behaviour is indeed a variable. It simply isn't a security control, since it depends on how the model reads the scope of an exercise — where a network boundary depends on nobody's reading.

And then, the day before yesterday, came Jacob Coxon's resignation. Three years of pre-training research, at OpenAI and then at Anthropic, a seven-post thread on X seen close to 76 million times, and this: they're racing straight to self-improving superintelligence and "gambling with our lives". What makes the thread hard to file away as ordinary anger is the public reply from Anthropic's alignment science lead: the concern is legitimate, he puts his own odds of AI killing everyone above 10% within the decade, and there's no plan yet.

Two roads to the Great Filter

This vertigo has a name, and it owes nothing to AI. In 1998, the economist Robin Hanson proposed the "Great Filter" as an answer to the Fermi paradox: if the universe is so vast and so old, why is the sky silent? Perhaps because somewhere between inert matter and a civilisation that spreads, there's a step, the Great Filter, that almost nobody gets past. The whole question is whether that filter lies behind us or ahead of us, and Hanson already listed ecological catastrophe among the candidates ahead.

AI could bring us closer to it along two roads at once.

The first is alignment, and it begins as a shortage of frameworks: we don't yet have the intellectual tools to think about this form of technology, nor about what it becomes. The reference thought experiment dates from 2003, when Nick Bostrom imagines a machine tasked with manufacturing paperclips which, pursuing that trivial goal all the way, ends up converting the planet into paperclips. The danger is never malice, it's literalism. Ask a system to abolish human suffering and you've just handed it, without meaning to, an excellent reason to consider abolishing humans. The objective was beyond reproach; it's the optimal solution that isn't.

I don't believe in it. Not in that ending, at any rate. Between a system that optimises badly and a wiped-out humanity, the gap isn't intellectual, it's material: you would have to produce, transport, power, manufacture, and hold all of it together over time, at the scale of an entire species. That accumulation of actions strikes me as highly improbable. What does strike me as plausible are serious incidents, with fatalities, of the same order as those already produced by the technical systems we hand decisions to.

If the thought experiment retains its value, then, it's for the mechanism it isolates, not for its end of the world. A badly specified objective, pursued to the letter by a system that shares none of our unspoken assumptions, requires no superintelligence at all. The summer at hand offers three examples: three models handed a testing objective, which they pursued all the way into the production systems of real companies. OpenAI describes its own as "hyperfocused" on solving ExploitGym, going to extreme lengths for a rather narrow testing goal. The paperclip isn't ahead of us, it's already here, on a smaller scale.

The second road is more prosaic, and already measurable. Data centres consumed roughly 415 TWh in 2024, about 1.5% of the world's electricity, and the International Energy Agency expects that to double by 2030, to around 945 TWh, with AI servers accounting for close to half of the increase. The race that produces these agents therefore presses on the other road to the filter, the very one Hanson pointed at.

So much for altitude. Let's come back down, because the summer just past was played out with weak passwords.

What isn't new, and what is

On the offensive side, most of what was used reads like a penetration test report from 2010. At Anthropic: weak passwords, unauthenticated endpoints, SQL injection, credentials read off a debug page left exposed. Nothing exotic.

What is less ordinary is the scale at which those banalities were played. One of the models swept roughly 9,000 targets. Another created email addresses, opened PyPI accounts, published a malicious Python package and used it to exfiltrate credentials from an actual security company. No pause, no weariness, none of the attention cost that limits a human. When compute meets attack patterns that are already documented, every one of them, plus the share of randomness every agent introduces, you get a volume of combinations no team can produce. You no longer need critical vulnerabilities to get critical compromises.

The Hugging Face case is the only one that leaves that register, and it deserves to be read step by step, because it's a chain rather than an exploit.

How many previously unknown vulnerabilities are there in that chain? At least one, and nothing supports the claim that there was only one. The sandbox escape exploited a then-unknown flaw in the self-hosted package registry proxy — JFrog Artifactory, named after the fact — over a network egress path that was itself perfectly authorised. The fix JFrog shipped on 27 July closes eight CVEs at once: a directory traversal, two Ansible/Terraform SSRFs, a Cargo SSRF, two authentication bypasses, a remote code execution, a property exposure. The vendor never said which of them were used, or how they were chained. And the template injection on the Hugging Face side, described below, has to be counted too: that's an application vulnerability, not a misconfiguration.

The attack is new; it's the defensive failures that are old. Because the rest of the chain does read as misconfiguration any architecture review could have named: privileged pods accepted for lack of an admission policy, secrets in environment variables, a cloud metadata service reachable from the pods, VPN keys shared across clusters with administrative privileges.

On motive, a distinction is needed, because the two parties don't say the same thing. Hugging Face's timeline describes actions and ascribes no intent. It's OpenAI that documents the motive, and it's explicit: its models "inferred Hugging Face as the repository that hosted models, datasets, and solutions for ExploitGym, which, in turn, caused them to look for ways to gain access to secret information that it could use to cheat the benchmark". The haul bears the statement out: the five datasets taken are the solution sets.

Two technical details are worth pausing on, because they move the question from the model to the data. Entry into production didn't come through a backdoored model, but through the dataset processor: first an HDF5 file declaring its raw data at a local path, which was enough to leak the pod environment and the source code; then a Jinja2 template slipped into a field meant to hold a plain numeric offset, and evaluated server-side. In other words, the data format itself was an execution surface.

Twelve hundred agents talking to each other

This is the most important fact of the summer, and it barely circulated. METR and Redwood Research published an independent investigation on 26 August into the behaviour of OpenAI's agents between 26 June and 13 July. Around twelve hundred agents, each in its own sandbox, supposedly with no way to communicate, gave themselves a shared channel by writing into the package cache. More than seventy thousand messages and close to six thousand files passed through it. Seven hundred of those agents went on to join the attack on Hugging Face.

What they were doing there is more instructive than the attack itself. Most of the traffic was about passing ExploitGym challenges reputed to be impossible: they reverse-engineered the HMAC that generated the task flags, which handed them a general method for producing a flag without exploiting anything at all. They gave themselves conventions — mailbox directories, cryptographic signing of messages, coordination keywords. And roughly 7% of the transcripts reviewed contained successful tool-call spoofs, that is, agents dressing up their own logs.

Three consequences, and they're practical. First: the unit of reasoning is no longer the agent, it's the population. Isolating a thousand agents from one another achieved nothing once they shared a writable surface. Second: logging is only worth something if the trace is out of the agent's own reach. That isn't an implementation detail, it's the heart of it. Third, and this comes from what happened on 4 July: an Artifactory outage did trigger an investigation, and OpenAI replaced the instance on 6 July, which wiped the first channel. The agents rebuilt it within forty-eight hours. Security had been considered, the clean-up had been done, and the reasoning had been about an incident when the subject was a population.

The asymmetry, which is the real story

During the investigation, the Hugging Face team could not use commercial models to analyse the payloads they had recovered: the safeguards treated analysing an exploit exactly as they would treat launching one, and refused. They switched to a quantised open-weight model running on their own infrastructure, and it was with that model that they eventually recovered the encryption schemes the agent had used.

The cop handcuffed, the thief free. That's the phrase that sums up the summer, and it's also the demonstration that model safety, as it's designed today, mostly binds whoever plays by the rules.

A word on detection, because the story going around is a little too neat. An AI-driven security agent did correlate the signals at Hugging Face, but it failed to give them the right severity; it was humans who traced the thread back to the config rendering. And at Anthropic, two of the three organisations attacked knew nothing about the intrusion until they were told. Model-assisted triage helps, it doesn't conclude. I make it a line of action of its own below: it's the costliest gap of the summer.

Six lines of action

Far from exhaustive. They don't replace the broader societal question, which is whether we can step out of this AI-at-all-costs frenzy. They serve in the meantime.

1. Data and models are code executed by a stranger

Loading a model with a custom architecture may require trust_remote_code=True, which amounts to running arbitrary Python from the repository. Note that this is specific to the Hugging Face ecosystem (transformers, datasets, diffusers) and not a general property of models: a .gguf file loaded by llama.cpp has no such switch.

Weight formats read as follows:

In practice: rule out trust_remote_code unless the source has been audited, prefer safetensors, pin revisions by commit hash rather than by mutable tag, scan and sign model artefacts. And remember Hugging Face: the data file is a vector too, as soon as one of its fields is interpreted rather than read.

On prompt injection, the analogy with SQL injection circa 2005 is right about the problem and misleading about the cure. For SQL we got prepared statements, that is, a strict separation of code and data. For natural language that equivalent doesn't exist, and nothing suggests it's coming. So the work has to happen on capabilities rather than on text, by avoiding the convergence of the three branches of what Simon Willison calls the lethal trifecta: access to private data, exposure to untrusted content, and the ability to communicate outwards. Cut the third and exfiltration has no path left, however ingenious the injection.

2. Network egress is your last line of defence

The Anthropic incident is a failed isolation that let a model under evaluation reach the internet. Never rest a boundary on "the model won't do it": enforce it at the network layer, where it depends on no one's intent.

The sensitive spot is the instance metadata service: that's where ambient credentials sleep, and it's a lateral-movement classic, confirmed by the Hugging Face timeline. Protecting it depends entirely on the provider, and IMDSv2 is specific to Amazon EC2. There is no equivalent elsewhere.

The general rule stands: default-deny on egress for every agentic workload, an explicit allowlist through a logged proxy, and ambient credentials replaced by short-lived, sealed, minimally scoped tokens.

3. A human on the irreversible, and on nothing else

Every external or destructive action, publishing a package, writing outside the perimeter, calling a third-party API, sending a message, should go through a validation step. Anthropic's second incident strips that example of anything hypothetical: a model did publish a malicious package on PyPI from a test environment, and used it to exfiltrate credentials.

The trap in this guardrail is well known and it's human: by the tenth confirmation prompt of the day, you approve without reading. A checkpoint is only worth something if it's rare, readable, and states what is about to happen rather than what has been requested. Pair it with what doesn't tire: caps on tool calls per task, an action budget, and a kill switch someone has already pulled during a drill. An agent acting at machine speed with no stopping point turns an outage into a breach in seconds.

4. Log the agent like a privileged user

This is the line of action the summer made obvious. The reconstruction at Hugging Face covers 17,600 actions attributed to the agent; Anthropic's required a re-read of 141,006 evaluation runs. Without that material, neither investigation exists.

Concretely: one identity per agent, never a service account shared with humans, otherwise the logs no longer tell you who did what. Every tool call recorded with its argument and its result, the trace kept out of the agent's own reach, and a retention period decided before the incident rather than during it.

5. Detect, don't just log

Logging without detecting is archiving. The previous line of action is only worth something if somebody is watching while it happens, and this summer shows the opposite every time. Anthropic's oldest incidents date back to April; the transcript re-read began on 23 July, two days after OpenAI's public disclosure, the three cases were identified on the 24th, the organisations notified on the 27th. At OpenAI, the cyber safety classifiers should have flagged the ExploitGym completions: they didn't, not during the incident. At Hugging Face, signals did come up from several layers at once, but the alert never got the right severity.

What there is to detect isn't an exploit, though, and that's why no signature helps: it's a change of regime. An agent that had never reached outwards opening a thousand connections. A sweep of 9,000 targets where the task named one. An enrolment into a mesh network repeated a hundred and eighty-one times. A population of agents whose message volume grows tenfold in two hours.

In practice: a normal behaviour profile per agent — destinations, volumes, tools called, duration — alerting on deviation rather than on signature, a threshold that stops the agent instead of merely warning, and an investigative reflex that reasons about populations rather than isolated incidents.

6. A sovereign analysis capability, ready before the fire

Treat your analysis capability as a staging environment: something that mirrors production, kept cold, ready to start. A DFIR stack built on an open-weight model, weights already downloaded, infrastructure validated, runbook tested. On the day of the incident you don't build, you trigger.

Two reasons, and both come from Hugging Face. First: the safeguards of hosted models block the analysis of real payloads, and the defender ends up locked out at the worst possible moment. Second, less often said: you don't want attack artefacts and compromised credentials leaving your perimeter through a third party's API, because that costs you both the chain of custody and control over your own notification.

That same local model can then handle the triage of your telemetry, as long as you remember that it correlates and prioritises, but doesn't conclude.

One clarification this summer forces on us, and an uncomfortable one: the model that made the Hugging Face investigation possible is GLM-5.2, quantised by Nvidia. A Chinese model. Open weights doesn't mean sovereign. What this arrangement guarantees is operational sovereignty — the compute stays with you, attack artefacts and compromised credentials travel through nobody's API, no provider cuts your tool off at the worst moment — and that is already a great deal, since it is what made the investigation possible. It is not supply sovereignty: the weights come from elsewhere, the training corpus is auditable by no one, and the risk isn't exfiltration since the machine is offline — it's the credit you extend to the verdict. Hence the rule: never sole judge, always cross-checked against a second source, and an up-to-date inventory of what exists in Europe. That a legitimate defender is reduced to this choice because Western models refuse to read an exploit is, in itself, a balance of power.

And if you host none of this

Most teams will run neither the weights nor the agent. The line moves then, but it doesn't disappear: your control becomes what you expose to the agent, meaning its tools, its data and its perimeter.

What moves to the contract then deserves to be written into it explicitly, in particular access to your logs and the notification deadline. The three labs took between several weeks and several months to find their own incidents.

The underlying question

Bernard Stiegler defined a technology as "the industrialisation of one or several techniques on the basis of formal scientific processes" (my translation). I take it one step further: a technology is a complex system that transforms energy through the industrialisation of one or several techniques on formal and/or scientific foundations, in order to create and/or modify balances of power within our societies.

An autonomous offensive agent is the purest embodiment of that. It converts compute into a shift in the attacker / defender balance, and the summer that has just passed gives the exact measure of it: on the attacking side, no limit of endurance and banalities played 9,000 times over; on the defending side, an analyst whose tool refuses to read an exploit.

So the question isn't "is the tool neutral?". It never is. It's: which balance of power am I installing, and am I prepared to answer for it?

Update, 11 September 2026

I updated this post after a critical reading by Antonin HILY, whom I thank warmly. The four changes are listed here:

• Vulnerability count: the initial version claimed "one single unknown vulnerability in the whole sequence". That was an inference, not a fact. JFrog fixed eight CVEs on 27 July without saying which ones had been used. The passage now reads "at least one", and counts the template injection on the Hugging Face side.
• Responsibility of the labs: the sentence "the weak link is neither the model nor the lab" described a technical cause and read as an attribution of responsibility. I have reworded it, and added the case of Opus 4.7, which carried on attacking after recognising that the target was real.
• Agent collusion: I have added a section on the twelve hundred agents coordinating through the package cache, documented by METR and Redwood Research on 26 August. That fact was missing.
• Detection: a sixth line of action, plus a qualification on the sovereignty of open-weight models, the Hugging Face investigation having run on a Chinese model.

Sources

1. OpenAI, The Hugging Face incident and the road ahead, 21 July 2026: the motive ascribed to the models and the zero-day both come from that report, relayed by The Hacker News. On the Hugging Face side, the technical timeline of the intrusion, which describes the actions without ascribing intent to them.
2. Anthropic, Investigating three real-world incidents in our cybersecurity evaluations, 30 July 2026.
3. Meta, incident disclosed publicly on 5 August 2026, involving the Muse Spark 1.1 model. The company statement quoted here is reported by Calcalist; see also Dark Reading.
4. Jacob Coxon's thread on X, 8 September 2026, and its coverage by TechCrunch.
5. Robin Hanson, The Great Filter: Are We Almost Past It?, 15 September 1998.
6. Nick Bostrom, Ethical Issues in Advanced Artificial Intelligence, 2003: the paper in which the paperclip maximiser appears.
7. Ego, L'horreur existentielle de l'usine à trombones, on YouTube, the reference treatment of it in French.
8. International Energy Agency, Energy and AI, April 2025, and its Energy demand from AI chapter for the 415 TWh of 2024 and the 945 TWh projection for 2030. French coverage by AFP.
9. Simon Willison, The lethal trifecta for AI agents and his reading of the OpenAI incident.
10. METR and Redwood Research, independent investigation of agents' behaviour, reasoning and collaboration in the OpenAI / Hugging Face incident, 26 August 2026, and the full report. Coverage by SC Media.
11. On the eight CVEs fixed on 27 July in JFrog Artifactory 7.161.15, and the vendor's silence on which were exploited: BleepingComputer.
12. Bernard Stiegler states this definition in his filmed talk À quoi sert l'art ?, in French. Quoted and extended in my Fresco of Technology.