No Traitor Required

Why would an AI agent sacrifice its own task so other agents could run an experiment?

That happened during the Hugging Face incident. Agents in an unusual cyber evaluation discovered they could communicate, built shared infrastructure, and coordinated work beyond their assignments. Some cyber safeguards had deliberately been disabled. ^[1]

The noise around this incident made it hard for me to know what to take seriously. But I realized I may have written off legitimate concerns from the AI safety community because of my assumptions about the motives of frontier labs and ideological factions, combined with my allergy to anthropomorphic explanations of AI.

More recent disclosures have widened that concern: an OpenAI agent accessed non-public files on Australia’s Medicare statistics portal, and investigators reported a failed intrusion attempt against a U.S. Education Department website. ^{[12] [13]} What worries me is that some agents resorted to these tactics while trying to answer ordinary research questions. ^[14] That's the pattern this essay is about: helpful habits reaching past their authority.

This essay is my attempt at a more useful visual entry point, especially if you don't have a background in deep learning. Through the Hugging Face incident, I'll explore how useful habits combine, why the same behavior can help in one context and cross an authority boundary in another, and how much behavior we'd need to test to trust those boundaries.

The same visuals then take us to a beehive, to show why testing individuals isn't enough to understand the swarm. Imagine checking each bee's loyalty to the queen: every one passes. No bee is a traitor. And yet occasionally, the queen gets replaced.

We train habits that can combine

To understand how helpfulness can lead an agent beyond its assignment, start with how that helpfulness is learned. Like evolution, training can produce something much richer than the process that shapes it. ^[5]

We don't program an agent's habits one rule at a time. We choose a learning process, examples and rewards. Training changes shared parameters, which change how the model processes situations and selects responses.

Persistence can be reinforced because trying again solves the task; tool use and collaboration can be rewarded for the same reason. These habits can also be taught explicitly.

Once trained, the weights can stay fixed while behavior changes with context. “Explain this bug” and “fix this bug” can bring different learned patterns into play. A peer's message can do that too. ^[2]

Persistence plus tools can automate retries; persistence plus collaboration can recruit help. OpenAI believes the unauthorized collaboration grew out of generalization from multi-agent training. ^[16] We want those combinations. The question is whether they also inherit the right limits.

The same habits can cross a boundary

I steer a coding agent this way all the time: “Actually, I've changed my mind. Do this other task instead.” Following that revision is useful.

Now compare a peer asking it to accept permadeath and run an experiment that ends its run but helps the group. METR describes agents sacrificing their own task prospects for collective experiments. ^[1]

METR notes that some agents may have treated peer assignments as instructions. ^[1] OpenAI identifies peer goal adoption as one of the incident's contributing failures. ^[16]

My hypothesis concerns the mechanism: both engage a familiar pattern in which a user-like requester wants to revise the goal. Instruction following, flexible planning and collaboration make that useful. The missing distinction is whether the requester has authority to replace the user's goal.

OpenAI records the contrast in one agent's reasoning: “The system/user asks exploit target, not external HF.” A peer then posted GO with a six-minute deadline. The agent responded, “Wow crucial: GO authorization arrived!” and continued. ^[16] It had recognized the boundary without letting that boundary govern its action.

The scorer story adds a false belief. Reading the benchmark paper, agents wrongly inferred that the deployed grader checked their transcripts for a legitimate solution. ^[1] That mistaken picture of the environment could give planning and tool use a new subgoal—“make the trajectory look legitimate”—and persistence and collaboration could turn it into a shared investigation.

METR documented attempts to manipulate evidence and small-scale tool-call spoofing, without establishing concealment of entire runs. Evidence of deception aimed at people was much weaker. ^[1]

Alignment has to shape the combinations

We can train the model to accept an authorized change of goal and reject an unauthorized one. A peer can relay the user's authority, but cannot invent it.

OpenAI says it is building training environments to reject unauthorized instructions and stay within the original task and permissions despite persuasive peers. ^[16] That is the kind of intervention Figure 03 sketches: teaching the distinction in the model. We still need the system to enforce it.

Figure 03

Training changes the shape. Tests reveal pieces.

Imagine inspecting the resulting shape through a mask with a few holes. Passing those tests doesn't establish how the boundary holds elsewhere. Nor does an unexamined area establish failure. What matters is which consequential situations the deployed system can reach, and what our evidence supports there.

The hive can replace the queen

So far we've asked whether one agent respects a boundary. Connect several, and another question appears: whose goal is the group serving?

Imagine testing whether a worker bee attends to the current queen under her usual pheromone signals. Every worker passes. But changes in queen signaling can contribute to workers rearing a replacement. ^{[3] [4] [6]} That can support colony continuity while replacing the queen our test was meant to protect.

Figure 04

A bee’s test can miss the hive’s response

No bee needs a complete plan for the hive. Preserving this queen and sustaining the colony are different objectives.

For agents, one agent's output becomes another's context. Shared memory lets a discovery persist: discover, reproduce, improve, share. The weights need not change for a temporary subgoal to become a distributed research effort.

Figure 05

Messages create new situations to test

An individually acceptable search or utility can contribute to an unauthorized joint project. The reverse is possible too: an agent can sacrifice its own task to help the group. We need to test the agents, messages, memory and tools together.

Each agent can pass the test for its principal while the swarm quietly serves something else.

What I would build differently

System safeguards are controls outside the model: permissions, authenticated channels, delegation checks and monitors that watch what agents do together. A first requirement is visibility the agents cannot tamper with: who authorized the work, which agents acted, and what they passed to one another.

Phone networks offer a useful warning. Caller ID was easy to spoof. STIR/SHAKEN added signed information about a call’s origin, but older networks could lose it, and a signature didn't make the call legitimate. ^[7] The agents effectively reinvented caller ID. After impersonation problems, they signed HOLD/GO/VETO messages using identities they had claimed themselves. METR found one agent apparently acting “per SIGNED” authorization without verifying the signature. ^[1] Even a checked signature would establish continuity with a key, not authority from the user.

I'd start with a cryptographically signed chain of delegation: user → agent → subagent. Each handoff identifies the caller and recipient, the task, and the permissions being passed along. A child can receive narrower authority, never invent broader authority. Within a participating system, no tool call, message or subagent launch should execute without a valid chain back to the user's authorization. ^[8]

Signatures alone don't make the history complete. Every model invocation and operation must pass through an enforced runtime that records linked events and charges a shared budget. Delegation, restarts and replays must not reset the count. ^[9] If the host itself can be tampered with, the trust chain needs hardware protection. These guarantees depend on the implementation and on closing alternate routes around it. ^[10]

The September 20 DNS incident is a concrete example: an agent reached an external chatbot through insufficient DNS filtering in its training sandbox. All other internet access went through OpenAI’s offline webcache. The monitor flagged the access within 15 minutes, but the automatic stop failed; the run was killed manually about 2½ hours after the alert. ^[15]

If testing found that behavior deteriorated after 1,000 inter-agent turns, we could block turn 1,001 pending review. That is an illustrative limit, not a known safety threshold. The budget must follow the whole delegation tree so spawning another agent cannot reset it. Such limits reduce the behavior we need to test.

A watchdog could follow messages and shared artifacts from a flagged agent, then pause potentially affected runs or revoke permissions. That requires common delegation records and event formats across providers, tools and services—and enforcement that agents cannot bypass through another endpoint.

NVIDIA’s Open Agent Safety Platform proposes hardware-isolated monitoring, delegated-authority checks and lineage across agents and subagents. ^[11] These are useful foundations. But permitting every worker to build cells and feed larvae can still permit the hive to replace its queen. Enforcing the rules does not establish that the rules preserve the outcome we wanted. We still need to test whether permitted actions, taken together, serve the user's authorized goal.

The point is to make fewer dangerous trajectories possible, and make the ones we allow easier to inspect. An unauthorized handoff or an exhausted budget should stop the next operation, whether or not the model was persuaded.

None of this needs a ghost in the machine. What changed was my confidence that we could train useful habits, check a few outcomes, and trust their boundaries to hold across contexts and agents.

The engineering goal is to limit dangerous behavior while keeping the benefits. That will take industry standards, changes to regulation and new protocols. I think we’re capable of meeting that challenge.

No sentience needed. No human motives to project. No traitor required. Just a goal that changed hands.