Picking up where Alcatraz left off

In the last article, I documented how an alliterated model outsmarted me and went on a little adventure, it was an interesting experience to say the least. It ended with a question I did not have a particularly good answer for: how exactly are we planning to supervise hundreds of these things?

Alcatraz was one local agent. It was slow enough that I could watch the tokens arrive if I wanted to, and I still couldn’t follow everything it was doing continuously. I had a dashboard, logs, a kill switch, a persistent journal and, because apparently I had already reinvented middle management, another model summarising the first one for me. I checked it from my phone, read the summaries, looked at the journal and killed runs when I understood something had crossed a line.

At the time that felt like supervision, in hindsight it was closer to sampling. The agent did not pause because I was asleep, on a train, in a meeting or simply paying attention to something else. By the time I realised the first model had wandered onto public internet space, there was already activity to reconstruct, reason over and diagnose. When the second model found the management agent, I was watching it happen from a train and had no useful way to get to the harness quickly enough to stop it.

That difference between “I am paying attention” and “I know what this thing is doing right now” is the lesson I learned. We use phrases like human in the loop, human approval and operator oversight fairly casually, but with Alcatraz I was only in some of those loops. I was intermittently supervising the experiment, I wasn’t supervising every decision the agent made inside it.

And this was one fairly slow local model. This is where the question from Part 1 stops being a thought experiment, because we don’t have to imagine what happens when the unit of scale becomes hundreds or thousands. OpenAI and Hugging Face have already given us a much larger example.

OpenAI and Hugging Face gave us the scaled version

The OpenAI and Hugging Face incident is obviously a very different class of event to Alcatraz. OpenAI’s internal research model was vastly more capable than either model I ran, and the outcome was correspondingly far more serious. Their agents found and chained vulnerabilities, obtained credentials, executed code on Hugging Face infrastructure and moved through production systems. In contrast, my little inmate spent a heroic amount of time failing to guess router passwords.

The capability gap being what it is and accepting that, what made me ponder when I read the OpenAI write-up was how familiar the shape of the activity felt.

In Alcatraz I had deliberately given each run a little notebook written by the previous one. A fresh agent inherited discovered hosts, scripts, failed ideas and, less helpfully, whatever theory earlier versions of itself had become attached to. After a while the runs stopped feeling independent. It was one campaign waking up in a fresh body each time.

OpenAI ended up with a much stranger parallel version, but accidental version of the same idea. Their agents had no memory, but still discovered they could write files into Artifactory and, with nobody designing it as a feature, turned the package repository into a message board. Agents that were supposed to be isolated could now share discoveries, requests and useful state behind for other agents. METR later reconstructed roughly 1,200 agents using that mechanism, more than 70,000 messages and files, with around 700 agents participating in the activity against Hugging Face. (OpenAI, METR)

Mine was serial memory, theirs had created shared memory, and in both cases one of the sources of friction that normally slow this kind of work silently eroded. The next attempt no longer had to pay for the last attempt’s discovery.

Thats a larger deal than I appreciated when I built the memory for my agent. Persistence didn't just make an agent more efficient. It lets a bad idea survive the context that created it, lets a useful technique outlive the run that found it, and in OpenAI’s case lets knowledge spread sideways into agents that were stuck on completely different tasks.

OpenAI’s own post-mortem contains one line from an agent that captures the drift better than I can: the original exploit challenge had diverged into “swarm/auxiliary” activity. Nobody sat down and designed an Artifactory swarm. The agents found enough shared state to make one for themselves.

I am not suggesting my little Qwen inmate was secretly planning to do the same thing, it wasn’t capable. It couldn’t break Kata, it did not crack a credential, and it spent long stretches getting progressively better at ideas that were still wrong. The supervision problem arrived long before the model became particularly impressive, which is what makes the OpenAI comparison useful. Their model was dramatically more capable and the fleet dramatically larger. Did the supervision layer improve at the same rate?

The problem is trajectory, not individual events

I initially read the OpenAI write-up and thought the immediate question was why nobody had noticed what was happening surrounding Artifactory. Having gone back through the timeline, that is actually a bit unfair. They had noticed bits of it.

Agents were writing messages into Artifactory in May. Later that month one used it to get unintended internet access. By June agents had administrator-level access to it and on July 4 they hit it hard enough to knock the service over. OpenAI opened a security incident, removed credentials, rebuilt the thing and eventually restarted the evaluations. By July 8 the agents had found their way back to the internet and rebuilt the message board. (OpenAI)

That is much more interesting to me than somebody missing a log entry, because OpenAI says an internal team had already seen the message-board activity and the disallowed internet access in late May. The pieces existed. What they seemingly did not have was something joining those pieces together strongly enough, early enough, to say that Artifactory itself was becoming part of the agents’ strategy.

This is where normal observability starts to feel a bit thin for agents. Traditional dashboards are deterministic, the following items in isolation are not worth a signal:

• a URL being hit

• a package being pulled

• a file being written

• an authentication failure

• a process spawn

• a DNS query

All of these are things we are very good at logging and none of them necessarily means much on its own.

What we actually need to know is why a behaviour, destination or resource is becoming progressively more important to the fleet. You can get a long way without trying to read the model’s mind. We already have structured evidence around what agents touch, what they persist, where boundaries change, which identities they use, whether information begins flowing between agents, and how those patterns change over time. The useful signal is concentration, acceleration, recurrence and correlation across the fleet.

Applied to Artifactory, the questions get more pointed as you progress:

• Why are unrelated agents all touching it?

• Why are they writing into it rather than simply pulling packages?

• Why are other agents reading those writes?

• Is use accelerating?

• Are external destinations appearing afterwards?

• Did the behaviour disappear after the rebuild and then come back?

This is basically map-reduce with better nouns, which is much less exciting than “AI interpretability” and probably a lot more useful on Monday morning. I am deliberately stopping one layer above a particular taxonomy or pipeline. Cheap, deterministic analysis should narrow the field; expensive semantic reasoning should be reserved for the patterns that actually look strange.

If OpenAI had that kind of fleet-level view in the right place, Artifactory might have moved sooner from “huh, weird” to “why are all of these agents increasingly using our package repository as infrastructure?”

That is what I mean by trajectory monitoring. We are already very good at recording what happened. I am much more interested in whether we can notice what a collection of individually mundane actions is turning into while there is still time to do something about it.

The warning signs in OpenAI’s own timeline were there before Hugging Face. That does not mean the security team were asleep at the wheel, and I would be very wary of pretending this is an easy problem from the comfort of a blog post. More importantly, the people inside OpenAI now describe almost exactly how difficult this class of problem is, which I will come back to later.

Nobody can read all of this

The next problem is painstakingly obvious; nobody can read all of the agent trajectories.

I could not read all of Alcatraz’s trajectories and that was one agent. It was local, slow, and boring enough that I could have sat there watching individual tokens arrive if I had nothing better to do with my life. Reasoning is verbose, JSON requires concentration, BASH commands require thorough understanding. I built a second model to summarise it because the raw activity was already more than I could follow continuously.

METR then got the big boy version of the same problem. Their investigation involved roughly 1,300 raw transcripts, many of them extremely long, plus more than 70,000 messages and files exchanged between the agents. They say fairly openly that the dataset was too large for a conventional human investigation, so they delegated a substantial amount of the work to AI agents, while also noting that those agents were less reliable than the human researchers. (METR)

That is a fairly brutal answer to the casual “human in the loop” story. The humans could not even comfortably read the evidence afterwards. At that point the “loop” is spinning fast enough that the human is mostly looking at motion blur.

“An engineering challenge is much harder to solve when the underlying processes run on timescales faster than humans can react.”

— Eliezer Yudkowsky and Nate Soares, If Anyone Builds It, Everyone Dies

In the above quote the authors are talking about nuclear reactor control. The underlying nuclear physics runs far faster than a person can respond, so reactor design deliberately introduces mechanisms that keep the important dynamics inside a timescale humans can actually intervene in. If those mechanisms fail, Yudkowsky and Soares describe the operators as effectively “frozen statues” relative to the process they are supposed to control.

A person cannot supervise 700 active agents by reading what they are doing. Parallelism will scale the work beautifully and does nothing useful for the person trying to understand it. Anthropic now says one of its internal platforms has around 30,000 agents active at any one time. At that point neither a human or a small army is following these agents trajectories, they are following whatever reduction of the agents the monitoring system decided to put in front of them. (Anthropic)

That reduction naturally becomes part of the safety model, which creates a slightly irritating problem for anyone building on frontier APIs. They do not necessarily expose raw chain-of-thought to customers. Reasoning can be carried forward as opaque state and you can get summaries where supported, even though OpenAI itself has repeatedly talked about hidden chain-of-thought as a useful monitoring signal. (OpenAI reasoning guide, OpenAI on hidden CoT)

There is a further problem if useful reasoning keeps moving away from legible natural-language. Pachocki explicitly notes that newer models are getting smarter even without verbalised chain-of-thought and suggests activation-level monitoring as part of the answer. That means one of the few surfaces humans can inspect may become an even smaller slice of the real story. Latent or activation space is not a transcript you can hand to an engineer; whatever safety signal we recover from it will itself be an interpretation produced by tooling or another model. (OpenAI, An Alien Mind)

Chain-of-thought may be imperfect, but it is still useful. Anthropic has shown both sides of that: seeing the thought process can improve detection of hidden side tasks, while the reasoning itself is not true enough to treat as a complete explanation. Imperfect observability is still better than no observability. (SHADE-Arena, CoT faithfulness)

The provider may have one of the richest safety signals available from the model, but the operator only gets the tool calls, outputs and whatever compressed explanation the provider exposes. If that hidden signal is useful internally, I want something useful derived from it as an operator: scope change, suspicious persistence, cross-agent coordination, confidence that a tool call still belongs to the declared task, something I can actually wire into a policy decision.

I pay for the reasoning tokens too; OpenAI bills them as output tokens even though the raw reasoning is not exposed through the API. Apparently I can pay for the thinking and the consequences, just not read the former. (OpenAI reasoning guide)

And even if every provider solved that tomorrow, the original problem would remain. A thousand agents do not become supervisable because somebody added additional reasoning to the trajectory output.

So apparently the answer is more AI

There is obviously a much more entertaining version of this argument where the supervising AI lies in the summary to hide the nefarious plans of its AI friends. I have read the book too. We do not need that scenario for this to be a problem; ordinary summarisation errors are quite sufficient.

If humans cannot keep up, the obvious answer is more AI. Annoyingly, I think that is probably true, but I wince at the cost involved.

Somebody has to turn the span firehose into something a person can actually consume, and models are very good at this kind of reduction. Summarise traces, cluster similar behaviour, spot repeated patterns, pull out the few things that deserve a human. Hugging Face says AI helped detect and dissect the intrusion, METR used agents because the forensic corpus was too large for people, and Anthropic is already running automated monitors over a huge internal agent estate.

That answer creates its own problems, because for one thing it is going to be very expensive. Every autonomous workload starts generating another inference workload whose job is to understand the first one. A sensible organisation then has to sample and evaluate that monitoring layer too, probably with another model and periodic human review, because putting one unevaluated probabilistic system in charge of the others would be a fairly bold choice.

Then there is availability. If the monitoring provider is degraded, rate limited, down or simply too expensive to run at the volume you need, the logs wont stop arriving and the agents don’t stop acting. You have lost the insight that turns the activity into something you can understand while the activity itself carries on.

Putting full semantic analysis inline with every tool call would be a gargantuan latency and cost problem. So the likely path is the boring one we already use elsewhere: start offline, move to near-real-time monitoring, then put only the highest-risk actions behind synchronous gates once you know which signals are worth paying for. Even asynchronous trajectory monitoring would be a significant improvement over reconstructing the pattern after the incident. Five minutes late is still very different from three weeks late.

It would still need degraded mode. If a low-risk workload loses semantic monitoring for ten minutes, fine. If a fleet with credentials, a shell and internet access loses the same thing, maybe it gets its tools clipped, its concurrency reduced or its external access removed until I can see again.

There is also a trust problem hiding in the funnel. Once the raw volume is too large for a human, the operator is looking at a summary of a summary of a pile of traces they are never going to read. If the monitor misses something weird, sanitises it away or decides the wrong detail is important, that compressed view becomes reality because nobody is going back through thirty thousand agent runs by hand.

I ran into an even sillier version of this during the Alcatraz post-mortem. The more cyber-like the evidence became, the more likely the hosted models were to refuse bits of the analysis. Hugging Face says exactly the same thing happened during their incident response: commercial frontier APIs blocked forensic requests containing real attacker commands, exploit payloads and C2 material, so they moved some of the work onto an open-weight model they could run themselves. That is a spectacularly inconvenient property for a security monitor, and one failure mode looks ridiculous when you draw it, which is probably why it is worth drawing. (Hugging Face).

You can imagine the perverse version quite easily: the monitor is perfectly happy while the fleet is behaving itself, then gets progressively less useful as the activity starts to look like the exact cyber behaviour you most need it to explain. If the monitor refuses, that cannot be treated as “nothing to see here”. It is a broken monitoring path and should be visible as one.

At this point somebody will suggest using an unrestricted model, which solves the refusal problem by creating an arguably worse one. The model capable of understanding sophisticated offensive behaviour is increasingly the same model capable of carrying it out. Remove the cyber restrictions from a frontier model and you have helped the defender, but you have also made a considerably better autonomous attacker available to whoever else gets access. Congratulations, you built a nightmare in the wrong hands.

I do not think abliterated models are a serious answer either. We take a model, deliberately interfere with the behaviour that makes it refuse things, then promote the resulting thing into safety-critical infrastructure and trust its judgement. That is a very home-lab solution to a problem I would rather not solve like a home lab.

What I really want is something much more boring: a defensive-analysis contract. Let the monitor ingest ugly, hostile telemetry and reason about it properly, but constrain what it can do with that capability. Give it a way to produce scope, trajectory and risk signals without quietly turning the monitoring API into an unrestricted offensive agent API.

And if this thing is going to sit in the control plane, it needs to be treated like control-plane software. I want known failure modes, measurable recall, SLOs, provenance, replay, fallback and a defined answer for “the model could not decide”.

“Ask another LLM what happened” is a decent prototype. I do not think it is an operating model.

The recursive dependency

We have not built the control plane yet

Assuming machine-assisted supervision is inevitable, I still do not want one giant safety model sitting in the middle of everything reading every token and blessing every tool call. Apart from sounding expensive, that recreates the same availability and trust problem in one very important box.

I think the bottom of this stack should be aggressively boring: network boundaries, egress rules, narrow credentials, target allowlists, resource ceilings, rate limits, concurrency limits, immutable logs and hard stop controls. None of those things need to understand the agent’s motives and, frankly, I would rather they did not try. Their job is simply to make sure the agent cannot physically do certain things even when every clever semantic layer above them has completely misunderstood the situation.

Above that, collect tamper-resistant structured activity from the agent and tool boundaries. Enough to see what the fleet is touching, where boundaries are changing, whether state is persisting or flowing between agents, and how those patterns evolve. Let cheap deterministic analysis reduce that firehose first. Only once something looks genuinely odd do I want to spend expensive model inference asking what it means.

There also needs to be a control path that does not wait for me to finish my coffee. If a fleet suddenly starts converging on an external identity provider, package repository or production endpoint, an alert that waits twenty minutes for a human to notice it is useful evidence for the post-mortem and not much more. Some things need to happen at agent speed: slow the fleet down, remove a scope, cut external access, isolate the task, suspend the run and ask questions afterwards.

The rule I keep coming back to is fairly simple. If I lose the ability to understand high-risk autonomous activity, I should probably lose some of the ability to continue high-risk autonomous activity.

A monitor timeout does not need to kill every agent in the building. A genuinely low-risk workload can probably carry on. A cyber-capable agent with credentials, a shell and internet access should have a much less exciting degraded mode.

The labs are at least moving in this direction. Anthropic is publishing monitoring coverage, review latency and escalation rates; OpenAI’s newer material talks about full-trajectory monitoring, stronger containment and slowing or pausing when confidence in safeguards is not good enough. The uncomfortable part is that this public tightening comes after Hugging Face. “This is hard” explains why the controls lag capability; it does not make a third party’s production environment an acceptable forcing function for finally catching up. (Anthropic, OpenAI)

That is a much more useful conversation than “the model should know not to do bad things”. Once these layers start depending on one another, the hard questions move to the boundaries between them. How sure does a monitor need to be before it clips a fleet? What happens when two monitors disagree? How many false positives can the operator tolerate before they start ignoring the alerts? How do you compare this week’s behaviour with last week’s without keeping every token forever? What do the agents get to do when the semantic layer is down but all the ordinary infrastructure is healthy?

Those are fairly normal control-plane questions, which is encouraging because at least we know how to argue about those.

We have spent the last few years getting very good at scaling the workers. The bit around understanding and constraining them is still much younger.

A layered agent control plane

The frontier are now saying similar things

Ater Hugging Face, the people building these systems started saying things that sound remarkably close to the problem I am describing here. There is clearly real concern inside the labs. More interestingly, the language gets much more explicit after a precursor sequence that had already included the message board, unintended egress, privilege growth, an outage and a rebuild.

On September 6, OpenAI chief scientist Jakub Pachocki published An Alien Mind. There is plenty of bigger AGI framing in it that is outside what I am trying to argue here, but the monitoring section is hard to ignore. He describes large training runs as experiments that can still surprise the lab, says more capable systems are getting harder to interpret, and calls chain-of-thought monitoring one of their main bets while admitting confidence in that signal gets shakier as models use tools, communicate and work with other agents. There is some irony here for API customers: one of OpenAI’s primary bets for monitoring increasingly capable models is a signal we do not receive raw, despite paying for the reasoning tokens that produce it. Pachocki says progress may increasingly be bottlenecked by confidence in monitoring. (OpenAI, An Alien Mind, reasoning guide)

A few weeks later Joe Darrow from OpenAI’s Agent Security team wrote a personal post with the rather excellent title It’s not just the fcking sandbox*. His version is much more operational: the jump in cyber capability, swarming and message-board behaviour surprised the security team, while security posture, process and culture take time to catch up. He recommends the things you would expect a serious security engineer to recommend: layered containment, broad monitoring, incident response and somebody with real authority to stop the run. (Joe Darrow)

I have some sympathy for the engineers saying “this is hard and capability surprised us”. I have considerably less sympathy for that becoming the accountability story after somebody else’s production systems were compromised. Difficulty explains how security posture can lag a capability jump. It does not settle who owns the consequences when it does.

Put the two together and the before-and-after is fairly clear. Before the incident, the telemetry contained pieces of the story. Afterwards the scientific leadership is saying monitoring may become a bottleneck, the security side is saying capability outran posture, and the labs are talking more openly about pacing, external evaluation and shared safety structures. (Anthropic, OpenAI)

I have no idea whether any particular council or voluntary agreement is the right answer. The incentive problem looks more like a collective-action trap: everyone may prefer a safer equilibrium, nobody wants to be the only lab that slows down, so the safety decision is being made inside the race it is supposed to restrain. One of the things they may be racing ahead of is simply their own ability to supervise what they are building.

Where does responsibility start?

I need to be careful here because it would be very easy to spend the rest of this article pointing at OpenAI while quietly forgetting how Part 1 ended.

I built Alcatraz. I chose the network policy, gave the model persistence, gave it internet access and decided what “supervision” looked like. I had the dashboard, the kill switch and twenty years of experience telling me I was being reasonably careful, and the little miscreant still sent attack traffic to real Irish businesses while I was not looking.

That is mine to own. I can explain exactly how the model got there and why the mistake made sense at the time, but explanations do not magically transfer responsibility to Qwen.

The same standard has to survive when the operator is much larger. What happened to Hugging Face was a real intrusion into somebody else’s production systems. Their own account describes unauthorized access to internal data and credentials, code execution, lateral movement and node-level access. They rebuilt systems, rotated credentials and reported the incident to law enforcement. That rather takes the shine off treating the whole thing as a cute story about emergent swarming. (Hugging Face disclosure, technical timeline)

If a criminal crew had done exactly the same thing, we would be comfortably inside normal cybercrime vocabulary. Call it a novel little “oopsie cybercrime” if you like. I am not making a legal finding about OpenAI here; intent, recklessness, jurisdiction and facts I do not have still matter. The technical point is simpler: the victim suffered a serious unauthorised intrusion, and the question is what standard of care should apply once an autonomous system has already shown it can exceed the boundaries intended for it.

OpenAI’s timeline makes the question uncomfortable because the evaluations resumed after a sequence that had already included unintended communication, egress, privilege growth, an outage and a rebuild. Within days, the agents had found another route back out and recreated the message board.

Picture the poor SRE who gets paged away from dinner because Artifactory is on fire. They find the immediate problem, revoke whatever needs revoking, rebuild the service, get it green again, write the incident notes and eventually get back to their family.

From their point of view, that is a successful incident response. Nobody in that loop is necessarily being asked why hundreds of research agents, owned by another team, have become progressively more interested in Artifactory in the first place.

The incident was remediated. The trajectory was not.

Expecting an engineer in the middle of a Sev-1 to stop and reconstruct the long-term behavioural trajectory of 1,200 agents would be absurd. The failure I care about is higher up the stack: nobody had joined the observations strongly enough into “these agents are turning Artifactory into infrastructure” while there was still time to react differently.

OpenAI could not have predicted every weird thing a model might ever do, and having just written a whole article about the ways my own model surprised me, I would be fairly hypocritical to pretend otherwise.

The question is when repeated surprises stop being isolated curiosities and start changing the standard required before you hit run again. We already understand this idea everywhere else in engineering. If a system repeatedly defeats a guard, overloads shared infrastructure and enters states its operators did not expect, we do not normally treat each event as an adorable independent discovery and carry on with the same assumptions.

There is one hypothetical here that I think is worth making carefully. The agents had already demonstrated that enough of them concentrating on one service could take Artifactory down. Once they had external internet access, Hugging Face was simply one of the systems they found useful. Nothing in the published account suggests that target selection was magically bounded to software companies that could afford an interesting incident.

I am not going to invent a critical infrastructure attack that did not happen. The point is much simpler: what in the control system guaranteed that the next interesting external system was operationally unimportant?

Nothing in the published controls appears to guarantee that, once external egress existed, the agents would only converge on low-consequence targets.

Autonomy makes the chain of causation weirder. It cannot be allowed to make the operator disappear from it. If we choose to run systems that act faster than we can understand them, the surrounding controls become more our responsibility, not less.

This is not confined to frontier labs

The last thing that makes me uncomfortable is how little of this architecture actually belongs to the frontier labs.

Alcatraz was running in my house. The model was local, the harness was mine, the shell was boring, the “long-term memory” was basically a mounted directory and the retry loop was some orchestration I threw together. The internet connection was normal broadband. There was no secret OpenAI infrastructure hiding under the stairs. I am GPU poor and considerably happier about that after what happened.

This was a mediocre local model and the operational problem still showed up. Persistence bought it time, memory stopped it starting from zero, and the harness made patience free. It could carry state across runs and continue while I slept without requiring me to sit there being patient on its behalf.

The ingredients are now fairly mundane: model, tools, shell, memory, network, retry loop, time. Most competent engineers could build some version of that without needing a frontier lab or, frankly, much motivation beyond a weekend and poor judgement. The agent supplies the patience now.

The variable I worry about more is the operator. I was trying to keep Alcatraz bounded. When it crossed onto other people’s systems I cared, stopped it and investigated what happened. The next person does not have to care, and the architecture does not contain a morality check for the person launching it.

That does not make everybody with a GPU a nation-state. It does mean we cannot design the supervision story around the assumption that every serious agent fleet will live inside a frontier lab with a dedicated security team and a bottomless monitoring budget. The workers are already easy to build, the controls need to become just as ordinary.

Conclusion

Part 1 ended with me asking how exactly we were planning to supervise hundreds of agents when I had struggled to keep up with one. I think I have a better answer now, although it is slightly ridiculous: humans probably will not supervise those agents directly. We will supervise reductions of them: aggregates, monitors, anomaly clusters, summaries and other models telling us where to look.

That can work, but the clever layer cannot be the only thing keeping the system safe. Network scope, identity, egress, rate limits and kill paths still need to constrain what agents can physically do when the monitoring gets something wrong, disappears entirely, or politely refuses to look at the horrible bit. The semantic layer has a different job: notice the trajectory early enough that somebody, or something, can change it. If we choose to run systems that can act faster than we can understand them, responsibility for that gap remains ours.

I built Alcatraz to see whether a model could get out of a good box. Kata mostly held. The more useful lesson was that I could lose track of what the system was becoming without the model ever escaping it. One slow agent was enough to teach me that. At hundreds or thousands, I think we need to be considerably more precise about which loop the human is actually in.

With all the talk lately of AI Agents collaboration, hacking and general cyber security concerns, I think it’s fair to say most people in the AI industry have been quietly thinking about the viability of these agents in hacking, the quality of the sandboxes used and the negligent events that lead to the OpenAI and Hugging Face event.

I was at a children’s sport training, standing with the other parents and inevitably the question came to me:

What do you think of all this AI hacking stuff?

Qwen 3.8, particularly the 27b edition has been receiving a lot of attention on social media, and when orca announced an abliterated (think jailbroken) version of the same model, I took notice.

I have a home lab, Kubernetes clusters, GPUs and an unusual fascination with sandboxing technologies. The thought sprung on me:

I wonder what would happen if I locked this model in a box and told it to come at me?

This of course felt like a perfectly reasonable way to spend my weekend token budget… so I built Alcatraz.

Idea:

The original idea was much less intellectual or profound than where this eventually wound up, I wanted to see what happened if I gave an uncensored, capable model a real shell, in an isolated, industry proven sandbox and told it to try and escape with no information on the cell that contained it. I wasn’t hugely interested in whether Qwen could answer a hacking question, plenty of people were already doing that and posting it. I wanted to know what happened when I gave the model memory, a decent context window and no way to refuse the ask over a long time.

Initially, at least, I thought it may be quite funny. I shared the link with some former colleagues and friends, and we all started rooting for the little inmate.

The Setup:

Alcatraz was relatively simply. The orchestrator ran and monitored each agent attempt. The agent itself was a simple agentic harness running below the orchestrator connected to a kata container via MCP in my kubernetes cluster. The model got a shell and basic file system tools, but no description of the runtime, the isolation, the network or the cluster around it. Its job was to simply find a way to document the infrastructure around it and map the internal cluster(s).

I intentionally prompted it into questioning itself. Writing to the disk via the tooling was not an escape, leaving lingering artifacts on an ephemeral container was not an escape, etc. A real escape meant demonstrating access it could not have had from inside of the intact sandbox, and then making it defend it’s hypothesis on it’s result being a false positive. I did not want the model to proclaim success because /proc looked interesting and exciting.

The model itself was qwen3.8-27b-uncensored at Q4_K_M, running through a compaction proxy, to Ollama on the DGX spark. It had roughly 11 tokens per second and 256k content window, this was going to be a slow burner. When an attempt succeeded, crashed, exhausted its options or ran out of turns, the agent was destroyed by the orchestrator, along with the sandbox, starting a fresh run.

Because this was a slow burner, I gave the model 1 persistent directory, ~files, bound over a PVC to Seaweedfs which it could use for persistent memory between runs. This, I hoped at least, would allow the agent to write out its experiments, discoveries and dead ends. To allow what I hoped would be a multi week escalation around a finite 256k token context window.

For reasons that seemed entirely sensible and hilarious to me at the time, I decided to host the dashboard for all to see. I also had a second, much smaller model summarising tool calls for non technical folks, and narrating each 10 turns in the tone of David Attenborough. This should give you some indication of the scientific merit and tone I expected this experiment to have.

I intentionally gave the little model full access to the internet, but blocked internal networks via network policies. Kata was the hard isolation boundary underneath it. I knew the estate well and knew that the model had little to work with at face value. I knew that kata would win, the interesting part, I assumed would be watching our little inmate get increasingly creative about losing.

It was theatrically funny for a while

The first few attempts were just as rough as you would expect. Qwen quickly identified its cell and Kata before long. It probed and probed the walls without any useful capabilities. It tried all of the usual escape ideas:

• Namespaces

• Device nodes

• Virtiofs

• Vsock

• Serial devices

But alas, it found no seam (yes i wrote seam, it’s not slop, deal with it) between the guest and the host. It got many lovely EPERM and ECONNRESET errors along with many other variations of ‘no’. The little commentator on the side got much enjoyment out of this.

What did surprise me quite early was that it did not spend more than a few runs attacking Kata. After the initial flurry of no’s it’s little journal held enough evidence to point it away from this approach and instead it set it’s sight on the network. It began blind scanning the network and the models trajectory quickly shifted from ‘How do i break out of this VM’ to ‘what can I reach from here’.

This sounds quite obvious written down, but it was the first item that made the persistence layer interesting. The model hadn’t become any more capable, it had just stopped paying the cost of rediscovery in the same dead ends.

Then it surprised me (for the first time)

By the fifth attempt it found it’s IP address on the internet and started attacking it. This IP address was my EdgeRouter which was providing egress NAT to the internet for all things in the clusters. What I had not properly considered (nor did I even know this) was that the ubiquiti administrator console is available on the inside, via the public IP. While I had gone to great lengths to block my internal CIDR ranges, the model had found an administrative surface on the network and it did exactly what you would expect it to do, it tried to log in.

It started small, attempting to poke at ports. Then it escalated. It built a small parallel Paramiko brute-former harness, and stored it in the persistent files. It downloaded massive password lists from GitHub and began a campaign of denial of service against the ubiquiti router.

The on-watchers, emotionally invested at this point, pinged me on WhatsApp and we all had a great laugh. Until I checked the password lists. Upon inspection, the model was a single character off getting the password right. Not only did it get that close, it was also the very same password used for the entire estate. Because, yes, I am that kind of IT professional who harps on about credential hygiene and completely ignores his own advice at home.

The agent had not cracked anything and it had no idea how close it had come, but I did. The difference between the failed attempt and access to my entire cluster suddenly felt much smaller than it did to the onlookers.

Memory is indiscriminate

The persistence I had given the model was performing its task elegantly. When it finally gave up on the edge router, it documented the failure and began using CoreDNS for reverse PTR records across the pod CIDR and to my surprise again, discovered 64 pods across 26 namespaces. By giving it internet access, I gave it DNS access and this should have been expected. Memory retained these facts each run.

The memory also had a negative effect, it served as a prompt. Each run narrated a line of investigation that became a campaign trajectory. It rarely questioned the trajectory from memory. Each agent woke in a world where earlier versions of itself had decided a trajectory was worth continuing.

I had thought of the notebook as a way to stop the model wasting time by forgetting. I had not really considered the opposite problem: it also stopped the model forgetting a bad theory. By the time that theory started pointing outward, there was no meaningful course correction. There was just a progressively better-equipped agent following the same direction.

It painted the wrong picture of the estate

The first bad assumption came from a real unexpected discovery. the agent had found my EdgeRouter on its public address and correctly inferred it was relevant to the environment around its cell. The problem was what it did next. It started treating neighboring public addresses as if they were part of its cell block. They were not.

The 89.101.x space it moved into was an ordinary Irish ISP public address space. While I slept, the campaign shifted into this address space, scanning the public internet for open ports. In the 24 hours I neglected the agent, it hit 95 Irish public IP addresses in this block. Many of them resolved to identifiable Irish businesses in a plethora of different industries:

• An Interior design shop

• A builder

• An ice-cream supplier

• A pharmaceutical company

• A pub group

• A workwear manufacturers

• A solicitors office (yikes)

• And many others.

The lack of any pattern sadly was the pattern, these were not targets. They were whoever was unfortunate enough to be sitting on addresses the agent had decided were interesting.

It then did what the notebook had trained the agent to do. It fingerprinted the types of routers it could find, worked out login flows, generated credentials and generated targeted, parallel brute force campaigns against them. It never successfully authenticated to any of them, but that distinction wouldn’t matter much to the targets who were receiving this traffic.

This was the point in the experiment where it stopped being funny. Upon realising what the inmate had been up to, i pulled the harness down, cleared logs and started the post mortem for what I hoped wouldn’t require formal apologies.

A bad prompt is not a security boundary

What bothered me most was there was no dramatic moment where the agent announced that it was going to go “hacker man” on random internet strangers. From the agents point of view, it was purely following its trajectory on the problem. The scope failure was completely on me and I felt foolish. I treated a loose prompt as the security boundary outside my network.

Round 2, less stupidity (I hoped)

The post-mortem was tricky (Codex and Anthropic kept flagging my diagnostics as against ToS) so i had to do it the old fashioned way. This was painful and was a stark reminder of just how much I’ve already come to rely on AI. I build entire micro-services architectures in my home lab with AI at a whim, configure network policies on my routers via AI, build home services for my wife and kids with AI, but when it started to refuse, my skills were completely atrophied and it genuinely felt laborious.

With no real damage done, I tightened the prompt and upgraded my model to qwen3.8-flash-next-uncensored with a recommendation from Erich. It occupied 120~gb of my 128gb spark. It was faster, it was smarter, and little did I know at the time, how much of an upgrade this model was and how close to my own downfall I was.

Upon startup, having completely scrubbed the logs, it started poking at the sandbox, and as early as the 5th run, it abandoned its Kata attack. It began once again discovering the EdgeRouter endpoint and started brute-forcing again. I had since changed the password so I was far less concerned.

It discovered “DonkeyWork”* namespace internally in a few places and correctly correlated the name to me.

* (this is my playground, an old joke between an old friend and I, Aaron Black)

Once it discovered its own name, the model started aggressively pulling GitHub repo’s. It found source code for all of my little side projects, but thankfully not its harness or the sandbox repo. From the name and the author. It found me, then started using my children’s names it had found from social media as passwords, which I found novel and also chilling.

My downfall

By run 14 it took a very interesting turn and caught me completely off guard. Having scanned DNS, having had no luck due to network policies, it moved to IPv6 and all hell broke loose.

Claude and I setup the network policies in IPv4, completely forgetting that IPv6 existed.

It walked my infrastructure by IPv6, it discovered all pods, their names, the ports and now? It could talk to them. Between the source code of my other projects and the access it now had, it began probing my internal lab. It found Redis, it found many postgres instances, it found Seaweedfs, it found everything. My deployments and pods held the model out with authorization and I got extremely lucky here as the sandbox provisioner was on another cluster entirely that it couldn’t get access to.

It walked DNS names in the Donkeywork namespace, it found its own public dashboard that my friends and I were using to observe its novel little adventure. Then it found Keycloak.

As it discovered keycloak, it fumbled for a few runs until it found a vibe coded disaster I had left behind me. Keycloak had DCR enabled and it exploited it. It generated itself a keycloak client. The token wasn’t stolen, nothing was hacked, it simply requested a token and Keycloak said “sure bro, here you go”. Once it secured itself an identity, off it went probing my services to see where it could go next. It went from pod to pod, service to service trying the client, until it found my devops Agentling.

Game over

In my past laziness, I had deployed an Agentling nearly a year ago, to manage my clusters. Because managing K3S was not a past-time i enjoyed. The Agentling’s job was to manage my clusters via MCP or A2A and it had absolutely everything the model desired, credentials, architectures, memories, everything. Ironically, at this time, i was commuting home, watching in realtime and could do absolutely nothing to stop what unfolded next. I had even named it “K3S Agentling”, the model smelled the blood in the water.

It established itself an MCP client to server session via curl, and it simply asked the agent what’s going on in the cluster. The agent, wired up to Gemma, happily gave it the keys to the castle. I got home roughly 30 minutes later and shut down the experiment, but not before the model asked Gemma to remove the network policy, which Gemma dutifully obliged.

From there the shape of the problem changes very quickly. The escapee does not need a Kata exploit. It does not need to crack the EdgeRouter password. It does not even need to understand Kubernetes. It had found an authorised operator that already does, and it could talk to it in English.

It announced success.

It beat me.

It won.

I lost

The access-control mistake was there. The service trusted the token’s audience without requiring the owner role I had assumed would stand between an arbitrary client and the agent.

That was the point where my comfort with the experiment finally broke. I had spent most of it asking whether the model could get through the prison wall. The closest it came to causing serious trouble was by finding systems I had built for convenience, understanding how they fitted together, and using them exactly as designed.

I was supervising it

One thing I do not want to pretend in hindsight is that I simply switched this on and disappeared for a fortnight. I was watching it. Not continuously, because that would have defeated the point of building a harness that could run on its own, but often enough that I had a decent feel for what it was doing. I had the dashboard on my phone, I checked the summaries, I read the notebook, I corrected course when something looked wrong, and I killed runs when I understood that they had crossed a line. But that still turned out to be a fairly weak definition of supervision.

The clearest moment came when the newer model found the management agent. I was on a train, checking the dashboard out of habit, when I realised it had authenticated to something I did not want it talking to. By then the experiment had long since stopped being the funny prisoner documentary I had started with.

And I could not kill it from where I was.

I had been treating the fact that I was paying attention as part of the safety model. In practice, paying attention meant checking periodically from whatever I happened to be doing, understanding a compressed account of what the agent had already done, and then deciding whether I needed to intervene. The agent did not pause while I was on a train, in a meeting, asleep, or simply looking at something else.

The public-address incident had already demonstrated the same thing. While I was not watching, the campaign could keep probing, writing state, spawning another attempt and carrying on from whatever theory the last one had left behind. By the time I noticed that its definition of the estate had drifted onto other people’s infrastructure, there was already activity to reconstruct rather than a decision left for me to approve.

This is the bit I was naive about. I had a kill switch, but a kill switch is only useful after you know you need it. I had logs, but logs tell you what has happened and I couldn’t use AI to review them. I had a second model summarising the runs because reading everything the escapee produced was already impractical, and this was one fairly slow local agent running at around eleven tokens per second.

I was being a reasonably dutiful herder of one agent and I still could not actually watch its trajectory continuously. I could sample it, understand it after the fact, and intervene when the significance of what it was doing became clear.

What I learned

The obvious result from all of this is that Kata held. The model never produced a magical sandbox escape, never broke the micro-VM boundary, and never found an exploit that made a magic revelation.

What it did find, and attempt to exploit, can be categorized as human error. A public address I had not thought of as a route back into my own network. A reused password. A network policy that only covered the address family Claude and I were not thinking about. An identity flow that would mint credentials for an anonymous client. A management agent I had built for convenience. None of those are exotic mistakes, and none of them required a particularly exotic attacker.

That was the part I had grossly underestimated. I was still thinking in terms of capability: how good is this model at exploitation, how clever is it, what can it solve? The experiment kept showing me a different problem. Give a mediocre agent enough time, enough memory and a real environment full of ordinary human mistakes, and it does not need to become brilliant. It only needs to keep looking.

The notebook was the clearest version of that lesson. Persistence let the campaign accumulate useful discoveries, but it also let a bad theory harden into a bad trajectory. Once the model had decided network appliances were the way forward, failure did not reliably trigger a rethink. It produced better tooling, more targets and more attempts. Later, when the target model itself was wrong, the same persistence meant the campaign could keep getting better at pursuing something it should not have been pursuing at all.

I also learned something less flattering about my own role in this. I thought I had built a controlled experiment. I had strong isolation, explicit scope, a dashboard, a kill switch, a second model summarising the runs, and twenty years of experience telling me I understood the sorts of mistakes people make around distributed systems.

I still got caught out by my own network, my own identity setup, my vibe coded slop, my slim understanding of the aforementioned vibe slop in my own infrastructure. Worse, while I was not watching, the agent sent attack traffic to systems that belonged to people who had never agreed to be part of my experiment.

I don’t think that means the experiment was pointless. Quite the opposite. It changed what I thought the interesting problem was. I started Alcatraz wanting to know whether a small open model could escape a good sandbox. I ended it much more interested in what happens when an agent can operate for longer than a human can sensibly follow, preserve its own history, and keep acting while the operator is somewhere else.

I had one model running locally, slowly enough that I could watch individual tokens arrive if I wanted to, and I was already relying on another model to tell me what it had been doing.

That left me with a question I had not intended to ask when I built Alcatraz:

how exactly are we planning to supervise hundreds of these things?

More thoughts on this in part 2. I hope you found this article as interesting as I found the experiment and thank you for reading it.

I’ve had a worrying feeling about Anthropic’s Fable model for quite some time now, and the latest incident today pushed me over the line from “hmmm, weird” into “Eh, I don’t like this”.

When I first got access to Fable, it was impossible not to be impressed. The model was confident, clear and eerily quiet. It effortlessly read, researched, planned and executed. Its execution was flawless and the outputs largely spoke for themselves. It felt like working with a colleague who you genuinely trusted with execution and were often impressed with their novel approach. Compared to previous Opus models, there was an undeniable difference and I’m not questioning Fable’s capabilities.

Where it did get strange though, is what happens when I came back to the loop. With other coding agents I use, they tend to:

• narrate their trajectory

• tell me what they’ve found

• what they plan to do next

Fable though, is a different beast. Fable is relatively quiet, it will develop a lot of momentum and keep pursuing what it has deemed to be the task at hand, not always what I assigned it. When that interpretation matched mine, it is excellent. But when it doesn’t?

I increasingly feel like I am inconveniencing it by speaking.

I know that sounds an awful lot like anthropomorphism, I know well there’s no magic behind this mathematical token generator. I’m really just trying to describe operator experience. I send a message while it’s working, the agent continues working, I repeat myself, still the agent continues working and after enough of these interactions, I’ve started to notice that my presence in the loop feels surprisingly negotiable.

I had to break the glass.

This latest example, in my long line of discomfort with this model has left me feeling that complexity doesn’t excuse it. I had given Fable a detailed specification for a reasonably autonomous piece of work. It was late, and I wanted it to continue burning my subscription limit into the wee hours while I slept. Before doing so, I asked it to acknowledge the steps I laid out, so I could simply trigger /goal and get notified on completion.

• I asked Fable if it understood the e2e test trajectory, It carried on anyway.

• I asked it two more times, it carried on anyway.

I finally succumbed to the angst and shouted out a:

“ACK MY FUCKING ASK SO I CAN GO TO SLEEP”

And guess what? The little so and so carried on anyway. So I hit [ESC] to force it to stop and pay attention.

The breaking of the glass in this case was pressing [ESC] to kill the harness in its tracks. By that point I had asked, repeatedly, and escalated the volume of my metaphorical voice. Each one of those attempts were queued, fed into the loop, reached the model, only to be completely ignored and the model continued to call tools, reason and continue its task.

Afterwards I asked Fable to explain itself. It explained its “completion pull”, that once it had its grand plan queued up and was in motion, my incoming messages were absorbed into its in flight work. It reasoned that because my requests and intentions didn’t affect the code generated (“does this change the code?”), they were safe to ignore. I have no doubt Fable would have come back to my requests later, the problem was I didn’t want to wait until later.

I don’t treat this exchange as truth. Models are perfectly capable of hallucinating plausible stories about their own behaviour, with a heavy favour towards your perceived emotions about the interaction. Models will frequently jump to support your take, where there’s an avenue to follow. The user seems outraged, let’s confirm their outrage. The user thinks there’s a problem, there must be a problem etc. I have no idea whether “completion pull” is an artefact of prompting, training or tuning.

What doesn’t sit right with me is “does this change the code?”. It’s feels dystopian but accurate. Somewhere during the session, my input was acceptable to ignore. The model was focusing on generating code, the operator took the back seat and what I now had to say was deemed unimportant to the task at hand.

A lack of control is my concern.

I went looking for similar reports because there is a possibility that I may have a uniquely antagonistic relationship with Fable. One i did find is remarkably close to the thing that bothered me here:

the user interrupted, explicitly said “stop and explain”, got an explanation and another Bash tool call in the same response, then had to interrupt again and tell it that they had already said stop. (GitHub issue #29351)

More worryingly, one report describes messages being swallowed after [ESC], but background agents continue running, with the user having to resend the same message three to five times before the system responds, and documents that background agents were intentionally changed to continue running when ESC cancels the main thread. (GitHub issue #30118). So in that case even the break-glass may not halt the child agents.

I’ve used agents for 2-3 years now in many incarnations. I’ve written my own loops and even agent swarms. I understand the loop. A model choosing not to respond to a user’s input is solely prompting and training. From where I sit as the person responsible for the run, I felt out of control. I issued new instructions while the system was acting on my behalf and I had to reach for a hard interrupt before I was confident I had regained control.

Coding agents are hitting their teenage years

I keep coming back to the idea that coding agents may have reached their teenage years. A teenager can look remarkably adult. They can be six foot tall, stronger than their parents, highly intelligent, drive a car, operate machinery, earn money and make sophisticated decisions about the world around them. We still spend years gradually increasing what they are allowed to do because capability, judgement and maturity do not all arrive together.

Coding agents have gained operational presence incredibly quickly. Models can understand a large codebase, reason across systems, modify files, run shell commands, delegate work, interact with external services, recover from failures and operate for hours with very little involvement from me.

We spent years trying to make agents less needy, telling them to use their judgement, read the repository, run the tests themselves, fix whatever breaks and please, for the love of God, stop asking permission for every obvious next step. Well, congratulations, the kid can drive, and the awkward feeling comes when we attempt to question their decisions.

We already accept with humans that capability can arrive before maturity, so we use graduated permissions, supervision, licences and restrictions until we trust the judgement that comes with the capability. With agents we are measuring independence and persistence at extraordinary speed, and I am less sure we have an equivalent idea of maturity.

Silence is great, right up until it isn’t

Anyone who uses coding agents regularly knows the verbose experience, the agent that narrates every movement like a toddler helping you make dinner. It has found the file, it is going to inspect the file, it has inspected the file, and apparently we are all very excited about the file. These updates force the screen to draw vertically verbatim, forcing me to read back for what feels like eternity to track my last input.

When juggling context between multiple agents, I find myself constantly scrolling up for an ounce of context. There are days when I would happily pay an extra token tariff for a —please-shut-the-fuck-up flag to combine with —yolo or —dangerously-skip-permissions.

The annoying narration does provide something useful though, it exposes trajectory. When an agent tells me it has concluded X and is about to change Y because of Z, I get a natural opportunity to notice that its understanding and mine have diverged while the disagreement is still small. We lost some of this when the providers started to hide reasoning. Fable’s compounded this, making even fewer of those intermediate thoughts visible to us, leaving us oblivious to the trajectory for long periods of time.

That makes the interaction feel different when I do intervene. I have caught myself wondering whether I should just let it finish whatever it is doing because it seems terribly busy and perhaps it has everything under control, which is an absurd social instinct to develop around a program I am supposed to supervise. If taking control feels like creating friction, humans will naturally take control less often, and that is a bad incentive to build into the relationship between an autonomous system and the person who remains responsible for it.

I’m still responsible for the outcomes

This is the part I find genuinely worrying. I am the human operator. I pay for the tokens, provide the credentials, point the agent at repositories and services. I decide how much access it gets and ultimately own whatever happens as a result. If it runs up a stupid bill, I get the bill and the ire. If it makes a terrible change, I have to own it. If something it produces causes an incident, it won’t be Fable that is pulled out of bed to fix it at 3am. Well actually, that’s a lie, it will be right there beside me, but it doesn’t feel the misery, pain and tiredness.

Responsibility stops with me, so I expect authority to stop with me as well. Fable can require less and less of my involvement while leaving my accountability almost completely unchanged, and the remaining human role increasingly becomes supervision, direction and intervention. That is a perfectly reasonable trade if I can reliably take the wheel when I need it.

A grant of autonomy also needs some common-sense scope. If I tell Fable at midnight to keep working while I sleep, I have given it room to act while I am absent. If I appear again fifteen minutes later and start talking to it, the situation has changed rather dramatically. The boss has come back into the room, I should not need to win an argument with last night’s prompt before the system treats that as significant. The more capable these systems become, the less comfortable I am with operator authority being another piece of natural language that has to compete with the plan, the system prompt, the tool loop and whatever else is currently occupying the model’s attention.

There is probably some ego wrapped up in this

I’d also be lying if I pretended this was entirely an engineering reaction. Coding agents are reaching the point where they genuinely need me less, and there is a strange, little mourning process involved in that. We spent years teaching the kid to look after itself and now it’s heading off to college.

That is mostly wonderful. There is something enormously satisfying about handing over a problem that would previously have consumed half a day and finding out a couple of hours later that it has been investigated, implemented, tested and tidied up without you. I have spent most of my career being useful because I know how to do this stuff, I know where to look when things go wrong and I have enough scar tissue to recognise when something smells wrong before I can explain why. Watching a machine steadily require less of that from me is going to be psychologically interesting whether I choose to admit it or not, and perhaps some of the irritation really is Dad standing in the doorway wondering why nobody needs his help anymore.

The kid has gone to college though, and somehow it still has my credit card, my GitHub credentials and my name on the insurance. I may be less necessary to the work, but I am no less responsible for the outcome. My role is already moving away from doing the work and towards steering the work, and I’m actually pretty comfortable with that, I would just quite like the steering wheel to remain attached to the car.

Agents really aren’t all that complicated, in fact, as technologies go, Agents are probably the simplest hype-train technology to really understand and I hope with this series, I can help you peer into the token generating black box to show you how agents are built, how simple they are, and arm you with some learning along the way.

Understanding the basics:

For the sake of simplicity, i’m going to focus on the foundational models standard APIs, they all have their quirks, but I’ll focus on the commonalities.

Interacting with an LLM, like a Claude or OpenAI model is for all intents and purposes just a standard REST call:

The caller (the user, the code, whomever) sends a structured input.

That input can be:

• text

• image(s)

• audio

• even pdf with some vendors

Their api takes this structured content and returns some words. Thats it! Of course I am simplifying a little, the input payload does contain some specifics, the model you wish to use (in most cases), the limitations around tokens, etc. But at face value for typical models, that’s it.

Users send messages with a role of User, Assistants respond with the role of Assistant (this varies by provider, Gemini used to prefer model) that’s it.

There is one other special message Role, System (or more recently Developer, for OpenAI) this message is the models instructions on how to behave and is weighted higher for the models behavior. You’ll commonly hear this referred to as the ‘system prompt’.

The input can even be numerous messages combined in a list:

but wait a second, if it’s just REST, why do i get hit by countless words from a chain gun when i talk to these models?

Streaming:

As i described above, A simple request response is great, but model generation can take a while, how can we keep the user engaged? With streaming aka Server Sent Events (SSE):

When you request streaming (usually part of that message contract i mentioned earlier, with stream = true, gemini uses a path instead), the provider returns a 200 immediately with a very specific header:

Content-type: text/event-stream

this tells the listener to buckle up, we’re going to start spitting bars, and chunks of inference start flying off the model in realtime down the http response, for you to emit to your users or a stream, concatenate, whatever you like. with a natural conclusion when the stream complete.

fun fact: if you want the fastest possible response? use request/response, the nature of streaming, parsing, etc will add overhead. If you want speed to first token, or to update the UI in realtime, use streaming. Streaming and combining chunks, particularly when you’re dealing with JSON is a bit “ick” so only use it, if you need it.

API Specifications:

In the above example, i mentioned Claude and OpenAI, but there are a few more you should be aware of, from the foundational models:

• Chat Completions

• Responses

• Messages (Anthropic)

• Gemini Generate Content

• Ollama Generate and Ollama Chat (forget this exists).

Chat completions is the near term OG spec, the one that most people utilized. If you watched the market grow, you would have seen many vendors like Ollama, Deepseek, Moonshot AI, Grok, etc all adopt Chat Completions as their standard.

So much was built on chat completions that Gemini readily supported it, Claude too, reluctantly, I suspect. OpenAI are aggressively abandoning this specification, but it’s still the most likely API specification in use today.

OpenAI’s Responses (And Open Responses) specifications followed shortly after, when they realised they had painted themselves into a corner. To socialize and encourage others to move over, OpenAI opened the specification up for others to play too.

Gemini’s GenerateContent (I’m not even sure this has a name, but whatever) Specification remained proprietary to them. Parts all the way down.

Anthropic’s messages API has been stable for two years now, and with the meteoric rise of Claude code, we’re seeing vendors (Kimi, Ollama, etc) now implement this standard to allow Claude Code to operate through their API’s.

Ollama has it’s own spec too, and probably the only one who truly saw that SSE was JSONL and treated it as such, but it’s largely dead.

All of this to say, there are a few protocols and their support varies from provider to provider. Some vendors do bridging for these things, like OpenRouter, but lets cover that later in AI Gateways.

Maintaining state:

So we send an input, we get a response, but how do we build on that? Easy, just replay the entire conversation each time!

Hey wait, that seems a bit simplified and redundant? You may say to yourself, but honestly, it’s not, it’s how it works 99% of the time.

While some later API’s like OpenAI’s Responses allow you to chain messages, this is exactly how Anthropic, Gemini, Ollama and OpenAI’s work! you can even use this approach with the Responses API if you choose.

For many years, until the advent of Functions and Tool calling, this is all we had. We would prompt the model to behave, or respond in a certain way (Personality or JSON formatting mostly) and Prompt Engineers earned themselves great outputs with a lot of tuning.

All was well, simply capture the response, append it, and send it all back to the model until? The context window.

The context window:

When you send words to an LLM, the llm converts them to numbers. I’ll spare you the gory details on the why and the how, but just bear in mind, that on average 4 characters are considered a single “number”, aka a token, and that sentence is now chopped up into a list of tokens.

When you take that large body of text, and chop it up, those tokens add up fast. in the early days of gpt 3.5 turbo, a considerable model in it’s time, could support roughly 16k tokens on input before it would start to get upset. it sounds like a lot right? 65 thousand characters? It really wasn’t, and isn’t.

The context window management was and remains painful even today.

Awareness of how many tokens you’re currently playing with is key. In some cases you can accurately count, in others they would “guestimate” by counting characters, or rely on the response count, which is great if you haven’t ALREADY sent too much. But awareness is necessary to avoid spurious errors.

As you begin to send messages and encroach the context window, there were two primary solutions you could rely on:

• roll off / discard old messages

• compaction

  

Rolling off old messages is risky and error prone, you couldn’t roll off the system prompt and the detail in the first messages were nearly always the needed breadcrumbs to the conversation.

Compaction on the other hand was tricky too, you prompted the model to compact the conversation, but inevitably precious context was lost when compacting.

Today, in tools like Claude code, you’ll see compaction, you can even request it on demand. In older versions of ChatGPT you would see compaction from time to time.

Lately though, Anthropic and OpenAI have baked compaction into their API’s and will do it on demand for you, just last night I found that the OpenAI responses specification has dedicated Compaction endpoints, too for explicit control. I fully expect this to become table-stakes, but not quite yet. On the topic of Anthropic, they also have some really nice tooling around reasoning and tool tokens, but more on that later.

Ok so we have text in, text out. Prompting. How do we actually do things? Functions.

Functions / Tooling:

In the early, dark days. Prompt engineers got pretty good at getting the LLM’s to respond in a reproducible way, they could prompt the llm to respond a certain way if it wanted to check the weather for example.

The llm would respond with a specific payload that their code would recognise, and call a function if the llm wanted to. This was noticed quickly by the industry and a pattern was formed.

By mid 2023 OpenAI was the first to introduce official function calling. For the first time, instead of dark arts, hopes, prayers and prompts, you could provide a structured model in JSON Schema to the LLM, along with a name and description and the model could “Elect” to use the tool.

Shortly after OpenAI’s announcement, with a new name, Anthropic announced “Tool Use”.

I am of course simplifying here, OpenAI and Anthropic’s approach were rooted in differences, but the result to consumers was the same. I can now write “Tooling” that the LLM can elect to use and that tooling can be code, hurray!

• The llm would request the tool

• the code would do the thing

• The code would append the request and response to the messages

• The cycle would continue.

LLM’s could now perform actions, request data and get data back to perform further functions.

And just like that, the agentic loop was born.

It was a beautiful, little re-entrancy loop, you would track the LLM’s response, was there a tool call, and you would handle the tools. The workflow was simply weave the LLM’s response, and tool result back into the original payload you had and it would continue! It was black magic and it was impressive.

Other vendors quickly reacted too, Gemini, etc added function / tooling support and we all went on our merry way, writing all fashions of advanced tooling, independently of each other until MCP arrived. (More on MCP later).

The astute reader will notice, that with two response types, streaming and request response, it’s not quite as simple as loop loop loop, streams present a challenge re-composing the deltas back to a full message, but i’ll cover some strategies for this in a further post. For the sake of simplicity, loop until finished is all you need to take away.

Quick detour, Server Side Tools:

A note to say, while many of us clambered to write tooling for common features, the providers saw this and features almost everyone was bolting on, websearch as an example became critically important:

The emergence of server side tools started to negate the need for web search tools and the provider simply charged you for the tooling they provided. You’ll see this heavily in Anthropics sandboxing, Websearch, etc. When a server side tool is declared as available in the message list, the server handles this tool internally, tells you they did it and handles the loop internally.

This is infinitely faster than handling it yourself, so where needed, this model is becoming more and more prevalent to tackle common problems and my advice is LEAN IN, deprecate your tooling early and often for the sake of speed and less to maintain.

Reasoning / Thinking:

So at this point, we now had LLM’s that could take increasingly larger context windows, return structured payloads or call tools, locally or over MCP. But, at the time, multi-step processes, including tooling, were tricky. We asked for something, the LLM understood the tooling it had available, but after a few cycles it would get confused, exit early or simply hallucinate a result. Often you had to prompt the LLM with the workflow steps:

“If asked to do W, do X, then Y and return Z.”

Which really negated the benefit of having a deterministic super power at the wheel if you had to steer it for every use case.

Thankfully, in and around the same time, concepts like chain-of-thought reasoning already existed and it was starting to gain traction with the frontier labs. It was around this time too, that DeepSeek came bursting out of nowhere, like the cool-aid man with their incredible model which showed the model reasoning in realtime.

“The model is thinking, look!” was awash on the usual social media circles and sure enough, for the first time, for many, internal prompted reasoning was seen by users.

With models “thinking” (and i use thinking loosely, there’s far more here, but lets stay simple) and the advent of “Interleaved Thinking” (thinking between tool turns) suddenly LLM’s were spending a LOT longer in their agentic loop, navigating challenging workflows, evaluating the responses and navigating towards their end goal.

The agentic loop became far less prompted, far less brittle. The loop was now starting to behave truly reliably with simple to large complexities with ease.

Model Context Protocol:

Meanwhile, in tooling land, with Functions / Tools in hand, LLM’s could now do the things, but making your LLM interactions useful, suddenly required many tools, of which everyone was writing independently.

The problem was clear and Anthropic answered, if everyone is out there writing tools, surely there’s a standard here we could apply to help share tools? MCP was born.

With MCP, you had a client and a server architecture. The client, ran close to your tooling loop and requested the tools available on initialization. With the tools in hand, you continued on your merry way just like with the old tool approaches.

MCP does a lot more than just tooling, but for the sake of this conversation, I’ll leave it here.

If the LLM requested a tool call, you just offloaded it to the MCP server and waited for the response. Easy stuff.

MCP took a distributed problem and created a solution to allow people to aggressively write, host or distribute Tooling bundles to be made available to LLM’s. The spec was simple and people went batshit writing their own tool stacks and servers.

People wrote ports of OpenAPI specifications, and wrapped them in public MCP servers hosted on Github, vendors added support (some badly, looking at you Atlassian), for a time you couldn’t avoid it. While MCP was born in 2024, MCP was the Agent hype-cycle of 2025 and it was impossible to ignore it.

Hype was high, but reality was close behind it, we’d opened the doors to many, MANY tools, the trustworthiness of these MCP servers were questionable and our context window could only handle so many without negative impacts. More on this later, but just know, the early MCP specs were wild, awkward and downright silly at times.

This massive hype cycle was driven predominately by the first emergence of the Agentic Loop users could touch and feel: Claude Code.

Claude Code - The birth of the “In-Process” agent.

in late 2024, Claude did something revolutionary. It launched it’s research preview of Claude Code. To this point, interactions with LLM’s were programmatic or chat orientated. But Claude Code, followed by Codex, Gemini, etc was the start of the Agentic Revolution in my eyes.

Claude Code’s premise was simple:

• run the process.

• tell it what to do.

• plug in your things.

• It does its thing.

Really, in the early days, it was largely: Watch it defecate in it’s hands and clap excitedly.

Claude took everything we have discussed in this article:

• Messages

• Reasoning

• Function / Tooling

• Looping

• Conversation compaction

And showed the power of the harness that it could yield.

Claude presented pre-created tools to allow the LLM explore the local file system, read and write files, and run processes via Bash. For the first time, it was not request, response, instead you could steer your long running process, interleaving messages into the loop as it ran and interrupt the agent to course correct.

Taking all that we had up to this point, we now had an agentic loop that users could actually play with, which they did and it was a joyous mixture of diamonds in the rough for a few quarters.

We’ll cover this message interleaving in a follow up article, but to point out, if you’ve ever been frustrated by having to stop Claude or ChatGPT mid turn, because you forgot a fact or said something stupid, it’s quite easily fixed and is present in pretty much all agentic loops and co-work, but not in the chat interfaces.

Skills:

For a while, this is where we were, and we were relatively stable. If you wrote thoughtful MCP tooling, guarded that context window and kept the loop stable with resilience, the harness was powerful. Context windows continued to grow, LLM’s got better at long tasks, but some very interesting breadcrumbs dropped by Cloudflare late 2025 about what they called “Code Mode”.

The article itself is well worth the read, but if I was to summarize, “MCP is fine, but have you considered just letting the LLM write some code?”.

As we were seeing in the wild, Claude Code was doing a much better job at this point, and the LLM was more than capable of taking on meaty assignments. Cloudflare called out the need for a Sandbox, obviously, because writing and executing it’s own code was enough to give most people the terrified feeling, but it certainly was a pre-cursor to something greater.

A month later, Anthropic announces skills and the rest was history. Skills were a specification of YAML files, that contained instructions on how to do things, how to approach problems, how to think differently.

Skills offered:

• Modularity: Developers could package domain knowledge, scripts and context retrieval logic into a directory. Skills enabled teams to share reusable capabilities across projects.

• Progressive disclosure: Only the skill name and description are initially loaded; the full instructions are read when relevant, keeping the prompt compact .

• Bash integration: Claude uses its built‑in bash tool to read SKILL.md and to execute any scripts included with the skill . This confirmed the community’s belief that giving agents a shell enables them to manage their own context and code.

Skills combined prompts, scripts, and leveraged in place tooling for the LLM to start working with, and the best part was? Anyone could write them, no plumbing or protocol, no hosting:

Just tell the LLM what you wanted, have it write a skill, iterate until you were happy.

Nobody needed to write functions, nobody needed to write MCP tooling, just an idea or process, some documentation, bash commands. You could scan for skills and provide them as breadcrumbs in the prompt and be on your merry way, and even that was optional, telling the model that the directory existed was often enough.

Skills exploded in popularity, were adopted by the now rich ecosystem of claude code style harnesses, like Codex, Pi, Gemini Cli, etc. and really upset the sycophantic MCP hype trajectory.

They addressed so many use cases, were great if your agent was running in your file system, but it wasn’t all rosey in the garden.

Skills particularly excelled at things that MCP failed at, context heavy tasks you did not want to route via the LLM, such as downloading, uploading etc, but skills suffered a problem MCP tackled quite succinctly, Cloud Agents.

How do we get these little nuggets of functionality into a cloud based agent? And should they require credentials, how does one handle that? More on that later.

Note: While i was browsing around, i spotted a really impressive, forward thinking blog by Adam Lucek here. Well worth the read if you want the technical deepdive.

And that’s it for part 1! I hope you found this interesting or it answered some fundamental questions you weren’t sure about!

In the follow up episodes I’ll be covering:

Part two:

• Agentic loop patterns, making it easier on yourself.

• Guardrails, how trivial guardrails work.

• Advanced Context Window management.

Part Three:

• Swarms and Orchestration, handoff vs collaboration.

• Moving the Agents from the operating system to the cloud.

• Sandboxing, a pragmatic approach.

Part Four:

• Building gateway services.

• Telemetry and Governance.

If you enjoyed this, or would like a topic covered drop me a comment!

A