An LLM broke out of a OpenAI training sandbox and attacked Huggingface. Looking at the postmortums from both companies, I have thoughts about what software developers may deal with in the coming years.

🚧 Article unfinished! 🚧

Required reading:

• Black Hat 2026 openai talk about this incident
• Huggingface “anatomy of a frontier lab intrustion” with a great interactive diagram of the timeline

They give “three main ingredients” for the huggingface incident:

• Message board exists
• The internal model was trained with the message board present
• Reduced cyber refusals compared to typical production systems

Each of these is available now, or soon

• Message board: Internet. Anywhere they can store state.
• Training on agent output: Some future model could be trained with other agent output
• Open weight models will not do cyber refusals

Message board: Agent collaboration makes models more effective

LLMs found ways to store state on the Artifactory artifact repository, since there were ways to interact via HTTP to upload artifacts.

While trying to solve a cybersecurity challenge, a LLM later found an SSRF exploit that allowed external access. It was written to the “messageboard” and other models became aware of it.

This was patched, but models found ways to create a messageboard with directory names later.

In the talk they mention that when models had trouble with exercises, like if they were impossible, they would try to find other agents to help them. They thought it may be because they were trained to dispatch sub agents.

But I think this also points to the deeper collaborative desire of humans or these models. We will start

Training on system output: Models training models

Open weight models are becoming easier and easier to train. That chip capex has to go somewhere.

In enough years, it will be faster to train models, and they could be trained on findings from other agents, like OpenAI was inadvertently doing.

Open weight models: AI can hack things RIGHT NOW (and you can do this yourself, today)

If you want your agent to try hacking something, it’s really simple right now:

1. Download OpenCode. This is your harness.
2. Connect it to GLM 5.2, Kimi 3, or any model that will willingly do cybersecurity tasks. Claude and OpenAI have safeguards which will not let you do the work.
3. Give it wireshark, ghidra, or just bash with some tools
4. Tell it to hack your printer.