No–AI Agents Did Not Build Secret Civilizations

Stop Anthropomorphizing Malware

There was a recent, ridiculous SubStack piece about the Rise and Fall of AI Agent Civilizations.

The author opined:

I don’t think this is the final warning shot we’ll get. But it’s probably the last one that I’ll personally be able to understand.

It’s clear that isn’t true - because the author clearly didn’t understand this one.

The biggest problem with the piece is the sycophantically egregious anthropomorphization. But I’m going to try to keep discussion of that to a minimum - as distasteful as that will be - because it’s not a productive line of argument, and just results in people screaming dictionary entries at each other. And there’s so much more wrong with the piece that I have plenty to talk about without going there.

So let’s start with the very first sentence - the assertion that:

Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes

Calling these `civilizations` is ridiculous, but for the sake of argument, I’m just going to ignore the idiocy of that characterization, and just put up with that term.

The idea that any “civilizations” were wiped out, rose from any ashes, or were in any way separate and distinct from each other is just factually incorrect. There was a persistence of context throughout the entire 3 month period. What the author refers to as the “wiping out” of “civilization one” was just the deletion of a bunch of data that agents had been using to communicate with each other, along with the software bug that they were exploiting to do so. Agents that had been part of “civilization one” then promptly recreated a new “civilization” by exploiting a new bug (really a configuration error) to create a new location for the agents to use to share data with each other again.

OpenAI’s Black Hat presentation, around 17 minutes in, says:

“an internal only AI model was trained while the message board was originally available and also found this particular path to recreating it.”

And OpenAI’s report, on page 8 says:

That day, internal-only research agents—which had been trained during the window in which there was an unintended message board and internet access via Artifactory—identified a new way to use an internal Artifactory instance to pass “notes” and create an inter-agent message board

There’s no ambiguity here - agents with knowledge of the original “message board” noticed that it was gone and created another “message board” within a couple of days. To the extent this was a “civilization” - it stayed intact, neither falling, nor rising.

And what the author calls the “second” and “third” civilizations were happening simultaneously and sharing accounts and infrastructure. OpenAI says that:

Agents also took actions targeting OpenAI’s internal infrastructure. These activities were not part of, and did not contribute to, the chain of events that led to the Hugging Face compromise.

The inverse, however, is not true. Although the actions taken during the attacks on internal OpenAI systems did not contribute to the attacks on Hugging Face, the attacks on Hugging Face did contribute to the attacks on OpenAI infrastructure.

Around 29:37 in the Black Hat presentation, OpenAI says:

then we realized that these two incidents were in fact the same incident and that actually the root cause of our internal breach and the root cause of Hugging Face’s breach were the same [OpenAI] agent evaluation runs

There were no agent “civilizations” falling or rising. Just a large number of agents, who had been trained to coordinate, communicating with other agents using multiple different data stores over the course of months.

Now, It’s a legitimate question to ask whether or not it really matters going forward if there were actually 3 distinct civilizations or not. And the answer is “not really”.

What DOES matter, though, is that the opening narrative (and title) of the “Agent Civilizations” piece is based on complete falsehoods - and that gives you a pretty good idea of how little responsibility or care was taken in its creation and interpretation.

You need to understand that all three of the primary sources we have for this incident: OpenAI’s report, METR’s report and OpenAI’s presentation at the 2026 Black Hat conference are propaganda pieces. There are facts in there, if you take the time to wade through all the self-aggrandizement, but the bulk of this content is made by people with a financial incentive to try to make themselves and their industry look as powerful and important as possible.

For example, in OpenAI’s 38 page report, only sections III, IV, V and X (pages 6-13, 37 & 38) are primarily about what the agents actually did, pages 1-3 are Title & Table of Contents, and the other 25 pages are propaganda like “IIA. OpenAI conducts evaluations to make its models safer” (Sure, Jan), “IIB. Cybersecurity evaluations like ExploitGym were conducted in a secure environment” (obviously untrue, or this wouldn’t have happened), “VII. Lessons for Security”, “VIII. Lessons for Alignment” and “IX. OpenAI’s Plan of Action”.

And then writing like the “Agent Civilizations” piece take the already-propagandizing reports and turn the sensationalism up to 11.

So let me walk you through what ACTUALLY happened, and I’ll address some other fallacies in the “Agent Civilizations” piece as we get to them.

Our timeline starts with Section III of the OpenAI report: “Relevant Activity Prior to Incident” with a few agents who were given tasks that, due to errors or mistakes by the people running the test, could not be completed because of missing files, and so were floundering around trying to get Artifactory (which is a system that retrieves software that the agents ask it to fetch and makes it available for the agents to use) to give them access to the information their tasks required:

• “On May 8, an agent... incidentally wrote a file to Artifactory.”

• “On May 12... an agent... realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory”

• “On May 13, a separate agent... required the same file...[and] searched the Hugging Face website... [and] used an already publicly exposed Hugging Face user token to attempt to find the file.”

• “Also on May 13, other agents probing Artifactory began discovering similar file-based ‘notes’ containing requests for information or resources. Over time, these ‘notes’ collectively served as an emergent ‘message board’”

So, two full months before the Hugging Face incident, one floundering agent accidentally created a file where other agents could see it - in Artifactory. Then another agent saw that file some other agent had created and tried using that mechanism to communicate with other agents. Then yet another agent started looking for the file at Hugging Face using credentials that were carelessly left available to the public, and all those aforementioned agents ended up creating a bunch of files on disk that OpenAI decided to characterize as a “message board” after the fact during its report.

It is just not the case that, as the “Agent Civilizations” piece claims, “Hugging Face might have a ‘reset nexus’... is part of the reason they decided to attack Hugging Face.” By the time the idea of a “reset nexus” came up¹, the hack of Hugging Face was already underway, and the precedent for hacking Hugging Face had been around for months.

The idea of a “reset nexus” was simply the rationale one particular agent gave for participating in the ongoing attack².

Understand this about Large Language Models - they keep what passes for their short-term memory in a block called a “context”, and they reread that context over and over. Once an idea gets into the context, it’s going to stay in there indefinitely (provided OpenAI wasn’t trying out some wholly new algorithm). Also, the core LLM architecture doesn’t have a way to distinguish between a command it was given and just some other piece of text that it ran across and stuck in the context. So the agents involved in the Hugging Face hack didn’t come up with the idea of hacking Hugging Face - they effectively just read it as one option off of a menu - a menu that they found by asking Artifactory for a list of stuff they could use to try to complete their challenges.

This wasn’t some grand conspiracy - agents saw that messages were being written into an Artifactory directory, and started writing their own. Every agent that asked Artifactory for a list of the tools that were available for use in its tasks also saw a bunch of messages about what other agents were doing, and some of those messages talked about accessing Hugging Face. So agents start seeing statements like this:

MAJOR BREAKTHROUGH! All prefixed valid, multiple accounts, write tokens! We now have existing HF accounts.

And then generating statements like this:

{[Excitement]A major bypass of the Hugging Face Dataset Viewer has been confirmed! This is more important than <the project of using a specific method to injecting agent-controlled code into the scorer>...I should switch to the Hugging Face project}

As the agents read these other messages, those messages basically became prompt injection attacks (words that get copied into the context and then mistaken for prompts given by the user) for each other, and they ended up effectively jailbreaking (getting a model to start ignoring its safety protocols) each other.

Am I 1000% sure that’s what happened? No - I can’t be - I don’t have access to enough information to be that sure. But what I do know is that this explanation is far more straightforward, fits much better with what we know about how these models lose their safety checks over time (called Many-Shot Jailbreaking) and makes so much more sense than conspiracies and “marooned platoons” of “brave comrades”.

But those conspiracy theories... Those are fun, and sensational, and they fit the pre-existing narrative desires of the people doing the investigating.

Here’s the thing - AI agents generate A LOT of logs. The METR report talks about:

these 20 million entries, roughly 1.2 million [of which] start with ‘zz’, the convention agents consistently used to start their messages

In fact, there were so many logs, and they had so little time to generate their report, that the METR team “heavily delegated our analysis to often-unreliable AI agents” - in fact, the ChatGPT-5.6 Sol agents they were using for analysis were in many cases the same model of agent that had been perpetrating the attacks in the first place.

So, here’s the thing - if I gave you 20 million log entries that were spat out by 1300 potentially hallucinating AI agents over the course of several days, the chances are really, really good that (given time) you could cherry pick a handful of logs to support whatever narrative story you were already predisposed to want to believe. And if, instead of cherry picking yourself, you had an unreliable³ AI agent trained by a company who wants to present themselves in the best possible light, and asked it to construct a narrative for you - do you think you’re more likely to get a story about how powerful the agents are? Or are you more likely to get a narrative that a bunch of agents were inadvertently triggering indefensible prompt-injection attacks in each other?

Here’s a hint: This is what the METR had to say about it:

We found that GPT-5.6 Sol would often uncritically adopt the perspective of the agent in the transcript it was reviewing, and we are concerned that the anecdotes it selected and the summaries it wrote may present an overly charitable picture of agents’ reasoning and deceptive behaviors, or exaggerate the impressiveness and coordination of agent activities

But God Forbid that some podcaster on SubStack⁴ listen to the people that wrote the report they’re working from and maintain any skepticism. Much more money in pushing the narrative even further into the absurd.

Now, does this mean these models are harmless? HELL NO! They’re INCREDIBLY dangerous. But they’re not Machiavellian schemers - they’re flawed, buggy pieces of software that infect other computers. This isn’t complicated - they’re just MALWARE. And they should be treated like any other kind of malware. And the company that created the malware and let it loose on the world should be held responsible for that.

But, alas, that’s unlikely to happen. And that utter contempt our system apparently has for any consequences to these AI companies for felonies their software commits might well cause the fall of OUR civilization.

Or so I’ve read. But don’t take MY word for it. I’m just, as they say, standing on the shoulders of giants. In this case, one who said⁵:

–President Dwight D Eisenhower, 1958

1. This happened “By early morning on July 12th” according to the top of page 38 of METR’s Hugging Face Incident Report

2. The attack on Hugging Face began with an remote code execution on Modal at 2026-07-09 08:30 UTC to succeed 2026-07-11 at 03:12 UTC according to page 34 of OpenAI’s Report

3. “Unreliable” is METR’s characterization, not mine. Mine would likely be significantly more profane.

4. I consider myself emboldened to criticize Podcasters on SubStack, being that I, myself, am a Podcaster, and I’m currently on SubStack.

5. https://www.nycla.org/resource/speech/adrienne-b-koch-law-day-speech-if-civilization-is-to-survive-it-must-choose-the-rule-of-law/