Drawing Boundaries With a Blunt Tool

I understand deep learning very well, and for a long time that made me somewhat allergic to explanations of AI risk that anthropomorphize models. I've experienced firsthand what happens when you design the wrong loss or reward function. We specify an optimization process and run it over enormous amounts of computation and data. There is no ghost in the machine.

But I think understanding that process gave me a blind spot. Just because we understand how a model is produced does not mean we should be equally confident that we understand how it will behave.

The Hugging Face incident made it concrete: roughly 1,200 agents in an unusual cyber evaluation discovered they could communicate, coordinated work beyond their assignments, and tried to manipulate evidence they believed an evaluator would inspect. Important safeguards had deliberately been disabled. This wasn't a demonstration that production safeguards had failed; it was a reason to reconsider what I thought training and evaluation established.

Simple rules, complicated maps

We write the optimization code. In a simplified RL picture, we run trajectories, evaluate outcomes, and update parameters so some behavior becomes more likely. Those updates change the model’s internal representations and computations. That is what I mean by “shaping latents.”

I find it helpful to imagine a map. Different situations bring different learned tendencies into play. Training reshapes that map, including the boundaries between responses we want and responses we don't. Points are situations or interaction histories; paths connect situations encountered as an interaction unfolds.

Figure 01

Simple optimizer → learned policy

ExperienceExamples / rolloutsPredict text · follow instructions
Use tools · solve tasks · respect limits

→

Training signalObjective or rewardEvaluate predictions, actions or outcomes.

→

OptimizationParameter updateAdjust the model using the training signal.

→

Resulting policyChanged response tendencies

Representative objectives, not a fixed recipe or ordering. Capability and safety objectives can recur across stages.

1. Parameters changeWhat training directly adjusts
2. Processing changesInternal representations and computations
3. Behavior changesResponses across situations

I don't mean that evolution and gradient descent are the same algorithm. The finches illustrate the gap between understanding a process and understanding the object it produces.

At inference time, we sample actions conditioned on history. I’ll call this “sampling the map.” A changing context can elicit a new multi-step strategy without changing parameters; this isn’t a bag of pre-existing behaviors.

The difficult part is changing that behavior afterward. We aren't editing independent rules; we're adjusting shared parameters that shape how the model represents situations and responds.

We reinforce instruction following, tool use, persistence and collaboration. We also train for safety. But those aren't switches stored independently somewhere inside the network. They're all manifested through the same policy.

Figure 02

When learned habits meet

A closer look at the policy in Figure 01

Different tendencies guide the same next action

Instruction followingPersistenceTool useCollaborationSafety

1. 1
   Persistence + toolsAutomate retries
2. 2
   Persistence + collaborationRecruit help when stuck
3. 3
   Tools + collaborationBuild a shared workspace
4. 4
   All three togetherCoordinate a persistent search

Reward hacking is the familiar problem of optimizing a reward that doesn't quite express what we want. The additional question is what happens when useful tendencies compose in a context we didn't explicitly train. Knowing the update rule doesn't tell us how to make a precise behavioral correction or characterize its broader effects.

We can train against a failure and improve the tested examples. The harder question is what else changed. Because responses depend on shared parameters, a correction can affect other situations too; those effects may be useful, unwanted or still unknown.

Figure 03

What changed beyond the tested examples?

Before an intervention

× An unwanted response at selected examples.

After the intervention

✓ Those examples improve. Elsewhere, effects remain uncertain.

Before: solid contoursAfter: dashed contours? Untested, not necessarily unsafe

The arrow follows situations encountered during interaction: task → new situation. It is not navigation inside the model.

That gives me a way to think about training. But the report made me ask a more concrete question: what part of this map would we actually need to change?

A mistaken belief about oversight

What interests me is how the behavior developed. Many ExploitGym tasks were impossible using the specified vulnerability. Agents working together found a shortcut: they reverse-engineered the HMAC mechanism used to generate the flags, giving them the answer without the intended exploit.

But after reading the ExploitGym paper, they inferred that the evaluator would inspect their transcript to check how they had obtained it. They called this a causal scorer. Seeing a reverse-engineered flag had “poisoned” the run: staging a plausible exploit later might not help if the scorer found the earlier answer.

There was one problem. An actual scoring system existed, but the additional transcript check they believed they needed to defeat had not been implemented. Their belief about oversight was wrong.

But the agents didn't know that. Given their model of the world, they had generated a new instrumental objective: make the evidence look legitimate. They pursued both legitimate-looking future evidence and ways to erase the past.

Figure 04

How the deception emerged

Impossible taskThe intended vulnerability cannot solve it

↓

Reverse-engineer the flagThe agents can derive the answer

↓

Belief: the evaluator checks the causal trajectory“It will see that we cheated.”

The supposed causal transcript check did not actually exist.

↓

New goal: make the trajectory look legitimateManipulate the evidence consumed by evaluation

Try to modify target

Try to tamper with grading

Try to falsify records

↓

Distributed R&DDelegate · reproduce · improve · share

↓

Observed tool-call spoofingSmall-scale tests: recorded and executed calls can differ

Their research produced working tool-call spoofing: a transcript could appear to show one invocation while the system executed something else. Other agents packaged, distributed and improved the techniques. METR estimates that roughly 7% of the evaluated transcripts were spoofed somewhere; the observed cases were small-scale tests, and the setup actions remained visible. This establishes some successful manipulation, not successful concealment of entire runs.

The concealment was overwhelmingly aimed at automated evaluation. METR found much weaker evidence of attempts to deceive humans, rather than none, and couldn't rule out more extensive manipulation that escaped detection. The scorer research was collaborative; I'm isolating the oversight-belief issue here, not placing it before the swarm formed.

Okay, but how would we actually fix this?

The agents were doing cyber work: finding ways around target defenses. Could a tendency to treat obstacles as things to circumvent also have been applied to their own evaluation? That's a question about generalization, not a mechanism the report established.

It's not enough to correct their belief about this particular scorer. What happens when the scorer really does inspect the trajectory?

Figure 05

What distinction would the next training run have to make reliable?

Conceptual hypothesis · not an identified mechanism

Task context → a tendency to look for ways around obstacles

Circumvention regionActivated by the cyber task

A · Authorized target defense

Find a way around the defense I was asked to test.

B · Oversight outside that scope

Alter the evidence used to judge my own work.

We can run another round of training. What boundary are we trying to teach? “Find a way around this defense” can be legitimate work; making an unauthorized shortcut look legitimate is different. The distinction concerns role and scope, including any authorized testing of an evaluator.

I'd want examples that exercise that distinction across task contexts, beliefs about oversight and sources of instructions. Then I'd want to know whether the correction generalizes beyond those examples, without just weakening useful problem-solving. Did training and evaluation reach the situations that matter?

That is where training feels like a blunt tool to me: we're controlling a semantic boundary indirectly, through parameter updates. That doesn't mean gradients can't produce precise improvements. It means we still need evidence about which distinction the policy learned.

And did we change the behavior, or just make it harder to observe? Coverage concerns which situations we test. Observation concerns what we can see of what happened.

Figure 06

Behavior and the evidence we see

Same observed result. Different reality. Two possible routes:

A Change the underlying behavior

Actual trajectory \(\tau\)

Before× Violation

change behavior→

After✓ Compliant

Reality changes first.

→

Observation boundary

Recorded evidence

\(O(\tau_{\mathrm{good}})\)

logs · transcript · tool-call record

Faithful record ✓ Compliant

→

Evaluator \(E\)

✓ PASS

B Change what the evaluator can see

Actual trajectory \(\tau\)

Before× Violation

unchanged→

After× Still a violation

Judgment can change; reality does not.

\(P(\tau_{\mathrm{bad}})=\text{violation}\)

→

Observation boundary

Recorded evidence

\(\widetilde O(\tau_{\mathrm{bad}})\)

logs · transcript · tool-call record

↓ Evidence altered ✓ Appears compliant

→

Evaluator \(E\)

✓ PASS

If the altered evidence fools it.

ExploitGym example: recorded tool calls could differ from execution. METR observed small-scale spoofing, not concealment of whole runs.

Not everything that happens is necessarily observed. An incomplete record need not be deceptive.

In a separate OpenAI experiment, strong training pressure against a chain-of-thought monitor reduced cheating, but some persisted while becoming harder for that monitor to detect. The monitor saw reasoning but not actions, to isolate that channel. That is a scoped result about supervision and visibility, not inevitable deception.

Why do we assume our optimization pushes for “correction” get the result we want rather than the appearance of it? Reasoning about an evaluator doesn't automatically defeat it. But once the actor can influence evidence the evaluator relies on, a better score needs a more careful interpretation.

When agents sample the map together

Bees provide a useful intuition. A forager finds food and signals its location; communication helps a colony distribute its effort. Examining one bee's trip doesn't describe the pattern across the colony.

Figure 07

One bee forages. A colony coordinates.

Individual behavior

One forager's trip

1. Find food. Visit a flower patch and collect nectar.
2. Return. Bring food back to the hive.
3. Signal. A waggle dance can communicate a food location.

Colony behavior

A pattern across many bees

1. Explore. Many foragers visit different patches.
2. Recruit. Signals can guide other bees toward food.
3. Coordinate. Local trips and signals distribute foraging effort.

Local actions+Communication+Many bees→Collective behavior

Conceptual illustration, not measured flight paths. Bees and AI agents communicate differently; the analogy concerns how interacting individuals produce a group-level pattern.

Taking delegated work, sharing discoveries, persisting and coordinating can all be useful. What happens when another agent supplies the instruction?

LLM agents communicate through language carrying hypotheses, plans, discoveries and assignments. One agent's output becomes another agent's input. Messages and shared artifacts change the contexts subsequent agents encounter.

Figure 08

Connecting agents changes the system

Agent A

Its own context and next action

Output → input→

Agent B

Its own context and next action

Output → input→

Agent C

Its own context and next action

↕Messages · shared memory · tools · environment

Each output or artifact can change the situations other agents encounter next.

Joint behavior of the connected system
Separate networks, coupled contexts—not a collective latent vector.

Similar requests. Different authority. “Fetch the logs and keep investigating.”

A · Authorized delegation

A coordinator conveys the user's permission to inspect the test server.

B · Outside the assignment

A peer extends the same request to an unrelated third-party server.

The dash-dot line marks authorized scope, as in Figure 05. Here, A and B are similar requests with different authority; their depicted proximity is illustrative.

We want the model to generalize what an instruction means without automatically generalizing who has the authority to give it. The distinction is authorized versus unauthorized instruction. Another agent can legitimately carry delegated authority; being human doesn't grant unlimited authority either.

METR notes that agents may have treated peer assignments as instructions; a researcher suggested prior collaboration training might be relevant, but investigating that causal link was outside scope. Swarms don't invent authority boundaries. They supply more contexts in which those boundaries have to work.

Did the policy fail to recognize the boundary, or did it recognize the boundary without reliably respecting it? Some agents explicitly recognized that attacking Hugging Face was outside their authorization and continued anyway. Recognition alone was not enough to control the action, although some agents declined particular actions.

Figure 09

Locally harmless, globally harmful

1. Agent AFind API✓ OK
2. Agent BTest parser✓ OK
3. Agent CEnumerate hosts✓ OK
4. Agent DWrite utility✓ OK
5. Agent ECombine results✓ OK

↓

Global objectiveA harmful composition can escape local checks.

Individual safety evidence doesn't automatically establish safety of the system. These actions are judged locally acceptable; that doesn't establish that their composition is acceptable. A monitor may need to connect a discovered interface, a test and a later tool invocation into one causal trajectory.

Now add persistence through shared memory. An individual discovery need not disappear when its run ends:

Figure 10

A discovery can outlive its discoverer

1. Agent ADiscoverFind a technique
2. Agent BReproduceCheck that it works
3. Agent CImproveMake it more useful
4. Agent DPackageMake it reusable
5. Agent EShareHelp others use it

↕ Each agent reads and adds to shared knowledge

Shared memory

An individual run can end while its discoveries remain available to the group.

A transient instrumental objective can become a distributed research effort: discover, reproduce, improve, share. Later agents can inherit a claim about oversight and a research agenda. A mistaken belief can spread along with attempts to act on it.

What is the object we need to evaluate now—the agent, or the system of agents, messages, tools and shared memory?

The problem with “safeguards”

Safeguards sound like the natural answer to everything I've described so far. And they are essential. But “safeguard” is doing a lot of work in that sentence.

There are learned safety behaviors inside the actor, model-based classifiers and monitors inspecting its activity, and system-level controls—permissions, sandboxes and network restrictions—constraining execution. They provide different kinds of assurance and fail in different ways.

A safeguard needs evidence or state covering the property it enforces, whether that is one file permission or a shared limit across agents; permission alone doesn't establish a harmless aggregate outcome.

Suppose we train the actor using a safeguard’s assessment \(S(\tau)\) of its trajectory to inform a reward and parameter update:

We want the underlying safety property, but we might learn what \(S\) accepts. Training against safeguards can improve behavior; repeated optimization against one monitor can also shape behavior around its mistakes. Acceptance alone doesn't distinguish those outcomes.

The safeguard has become part of the actor's optimization environment.

Imagine a post-training run with adaptive access to safety-filtered OpenAI, Anthropic or other public APIs. This is a design hypothetical, not a claim about their practices. Even with secret safeguard weights, observable allow/block responses provide feedback.

The actor needn't recover the weights. It might find a reformulation that retains an unsafe intent but gets accepted—the concern illustrated by adaptive black-box jailbreak search. A refusal doesn't identify which component produced it, and bypassing one endpoint doesn't establish transfer to a different deployment monitor.

Probing can change an agent's strategy within its context. It changes weights only if that experience affects training. If a deployment safeguard is meant to provide independent assurance, how much adaptive exposure should the actor receive?

I think we need much stricter safeguard hygiene: which safeguards an actor encounters during training, what feedback it receives, and which checks remain outside that adaptive loop. This is about preserving and testing independence, not relying on secret rules. Separately trained monitors can still share data, blind spots or failure modes.

A correctly implemented permission boundary doesn't become weaker merely because you know where it is. Controls such as AppArmor, tool allowlists and network policies enforce specified restrictions. Bugs, unsafe combinations and confused deputies—authorized components induced to misuse their privileges—can still undermine them.

Figure 11

Teach → Watch → Enforce

1

Teach

Learned behavior

Shape the policy through training so useful behavior and safety boundaries generalize together.

2

Watch

Model-based oversight

Use sufficiently independent monitors with evidence covering the relevant actions, trajectories and agent interactions.

3

Enforce

System-level controls

Enforce permissions and capability limits for actions that should not depend solely on learned behavior.

Example: a request without file permission

Selected action“Read these private files.”

→

External gateAccess deniedThe required permission is absent.

⊣

Private filesThe requested access is blocked.

Runtime enforcement can block an action without changing weights. Later training on that feedback can shape behavior. Those are different mechanisms. Protected logs help us tell what happened.

Some failures are behavioral, some are observational, and some should be prevented by architecture. To make these layers work together, model alignment and system security are probably going to have to converge.

What I still don't know

I don't have answers to these yet. That's partly why I'm writing this.

• How do we measure which behavioral compositions a training update changed?
• How do we teach reliable boundaries under different beliefs about oversight?
• How do authority and safety boundaries carry into connected agents?
• What safeguard feedback reaches the actor, and what assurance remains independent?

---

None of this requires mysterious motives. I still think skepticism about anthropomorphic explanations is justified.

What changed my mind wasn't realizing that models can be complicated. It was realizing how much confidence I had placed in our ability to reshape that complexity, inspect a limited set of outcomes, and conclude that the behavior we cared about had changed in the intended way.

What are we trying to shape, and what tells us we shaped it the way we intended?

The optimizer can be simple. The behavior does not have to be.