Loop engineering

The agent worked when I demoed it and then failed a week later in front of people, on a task it had already completed correctly twice. Nobody grades you on your average when they are watching the run happen. For two months I responded by fixing the wrong layer, rewriting the system prompt every time the agent did something stupid, and every rewrite fixed the failing case while quietly breaking a different one. What I was short on was never prompt quality or model capability, it was consistency, and consistency is a property of the loop.

Why prompt iteration stops paying

The number that finally reframed this for me comes from tau-bench, the tool-agent-user benchmark Sierra published in June 2024. Alongside the usual pass@1 they report pass^k, the fraction of tasks where all k independent attempts succeed. Not "at least one", all of them.

State-of-the-art function-calling agents of the gpt-4o class solve under 50% of tasks on a single attempt. Run the same task eight times and require every run to succeed, and pass^8 in the retail domain drops under 25%. Same model, same task, same prompt. The only variable is that you ran it more than once.

That gap between pass@1 and pass^8 is the entire argument for treating the loop as the unit of engineering. A demo that works once tells you almost nothing about whether the system works. If you have ever committed to a date on the strength of a good screen recording, that number should sting a little. I had done exactly that, and the failing week was the bill arriving.

Prompt iteration optimises the mean. What kills you in production is the variance, and variance is a property of the loop, not of the string you pass in.

The loop shape worth copying

Anthropic describes the agent loop as four stages: gather context, take action, verify work, repeat. The stage most homegrown loops skip entirely is verification, and it's the one that converts a stochastic process into something that trends toward correct.

They name three mechanisms for it. Rules-based feedback, which is linting, type checking, and test runs. Visual feedback, meaning screenshots the model can look at. And LLM-as-judge for the fuzzy rules that no linter expresses. Their own caveat on the third is worth quoting because most people skip it: it carries latency tradeoffs and "isn't considered very robust". So the ladder has an order. Deterministic checks first, judge last, and only for things a deterministic check genuinely cannot express.

There's a nice second-order point in their guidance about rules-based feedback: TypeScript with linting gives an agent more feedback layers than raw JavaScript does. The language choice is a loop design decision. More places for the environment to say "no, that's wrong" means more chances for the agent to correct itself without a human in the path.

# loop.py
from dataclasses import dataclass, field
from typing import Callable, Literal
 
Verdict = Literal["pass", "fail", "unknown"]
 
 
@dataclass
class Check:
    """One rung on the verification ladder.
 
    `cost` is a rough ordering hint, not a measurement. Cheap deterministic
    checks run first so the expensive fuzzy ones only see work that already
    survived the mechanical ones.
    """
 
    name: str
    cost: int
    run: Callable[[str], Verdict]
 
 
@dataclass
class Loop:
    """Gather context, take action, verify, repeat."""
 
    checks: list[Check] = field(default_factory=list)
 
    def verify(self, artifact: str) -> tuple[Verdict, list[str]]:
        """Run checks cheapest-first and stop at the first hard failure.
 
        Returns the verdict plus every check name that ran, so the failure
        that stopped the ladder is recoverable from the trace.
        """
        ran: list[str] = []
        for check in sorted(self.checks, key=lambda c: c.cost):
            ran.append(check.name)
            verdict = check.run(artifact)
            if verdict == "fail":
                return "fail", ran
        return "pass", ran
 
    def step(self, context: str) -> tuple[str, Verdict]:
        """One full turn of the loop."""
        artifact = self.act(context)
        verdict, _ = self.verify(artifact)
        return artifact, verdict

The ordering matters for cost as much as for quality. A judge call on an artifact that would have failed tsc is money you set on fire.

Context is a budget, and it's smaller than the context window

Anthropic frames context engineering as "the set of strategies for curating and maintaining the optimal set of tokens during LLM inference", and the operative idea is the attention budget: it's finite, it has diminishing marginal returns, and the goal is "the smallest possible set of high-signal tokens". A million-token window is not a million tokens of usable attention.

Chroma's context rot report from July 2025 is the measurement behind that. They tested 18 models, including Claude Opus 4 and Sonnet 4, o3, GPT-4.1, Gemini 2.5 Pro, and Qwen3, and found performance degrades consistently as input length grows even on trivial tasks. Not hard reasoning tasks. Trivial ones. When the semantic similarity between the needle and the question is low, degradation is faster. A single distractor in the haystack hurts, and four distractors compound the damage.

The result I keep coming back to is the strangest one. Models did better on shuffled haystacks than on logically structured ones, and this held across all 18 models. Coherent surrounding text hurts retrieval compared with incoherent surrounding text. I don't have a clean mechanistic story for why, and I'm suspicious of anyone who offers one confidently, but it does undercut the intuition that neatly organised context is automatically better context.

The cleanest argument for compaction I've seen is from LongMemEval, where a focused prompt of roughly 300 tokens was compared against a full prompt of roughly 113K tokens, with a large gap in favour of the focused one. Three hundred tokens against a hundred and thirteen thousand. That is the whole case for curation in one comparison.

Compaction in practice means summarising the conversation as it approaches the window limit and reinitialising with the summary. What Claude Code preserves through that is instructive: architectural decisions, unresolved bugs, and implementation details, while discarding redundant tool outputs. The reinitialised context is the summary plus the five most recently accessed files.

# compaction.py
from dataclasses import dataclass
 
KEEP_RECENT_FILES = 5
 
 
@dataclass
class Transcript:
    """A conversation plus the files it touched, in access order."""
 
    messages: list[dict]
    files_touched: list[str]
 
 
def compact(transcript: Transcript, summarise) -> Transcript:
    """Summarise a near-full transcript and reinitialise from the summary.
 
    Recall first: the summariser is instructed to over-include. Precision
    comes later, by iterating on the prompt once you can see what a dropped
    detail actually costs downstream.
    """
    summary = summarise(
        transcript.messages,
        keep=["architectural decisions", "unresolved bugs", "implementation details"],
        drop=["redundant tool outputs"],
    )
    recent = transcript.files_touched[-KEEP_RECENT_FILES:]
    return Transcript(
        messages=[{"role": "user", "content": summary}],
        files_touched=recent,
    )
 
 
def clear_tool_results(messages: list[dict], keep_last: int) -> list[dict]:
    """The lightest-touch alternative to full compaction.
 
    Blanking stale tool results reclaims most of the space with none of the
    summarisation risk, because nothing gets paraphrased.
    """
    seen = 0
    out = []
    for msg in reversed(messages):
        if msg.get("role") == "tool":
            seen += 1
            if seen > keep_last:
                msg = {**msg, "content": "[cleared]"}
        out.append(msg)
    return list(reversed(out))

The tuning rule Anthropic gives for compaction prompts is the one I'd have gotten backwards on my own: maximise recall first, then iterate toward precision. Dropping something important is much more expensive than carrying something redundant, so start greedy and trim later. And before you reach for compaction at all, tool result clearing is the lightest-touch version of the same idea, since it reclaims space without paraphrasing anything.

Sub-agents are the other lever. A sub-agent can burn tens of thousands of tokens exploring and return a distilled summary of often 1,000 to 2,000 tokens. The search happened, the parent just never has to carry the transcript of it.

Budgets and termination, using mechanisms that exist

Every agent framework ships a turn cap, and every one of them is a blunt instrument that you still need.

The OpenAI Agents SDK has max_turns, which raises MaxTurnsExceeded. LangGraph has recursion_limit, defaulting to 1000 super-steps, which raises GraphRecursionError, plus the managed RemainingSteps channel that lets a node see its remaining budget and wrap up deliberately rather than dying at the wall. Guardrail tripwires act as out-of-band terminators, killing a run on a condition that has nothing to do with turn count.

There's an implicit budget most people never think about: tool output size. Claude Code truncates tool responses at 25,000 tokens by default, with pagination and filtering as the recommended pattern rather than dumping everything and hoping. If your tool can return a hundred thousand tokens of log, the truncation boundary is now part of your loop's semantics whether you designed it or not.

Verbosity is a knob too. Anthropic's Slack example shows a ResponseFormat enum where the concise setting produced 72 tokens against the detailed setting's 206, roughly a third. Exposing that as a parameter the agent sets per call, rather than a global default, is close to free.

# budget.py
from dataclasses import dataclass
from enum import Enum
 
TOOL_OUTPUT_LIMIT = 25_000
 
 
class ResponseFormat(str, Enum):
    """Verbosity as an explicit, per-call parameter."""
 
    CONCISE = "concise"
    DETAILED = "detailed"
 
 
@dataclass
class Budget:
    """Turn and token caps, tracked together.
 
    Turns alone are a bad proxy: one turn that reads four files costs far
    more than one turn that writes a line of code.
    """
 
    max_turns: int
    max_tokens: int
    turns_used: int = 0
    tokens_used: int = 0
 
    @property
    def remaining_turns(self) -> int:
        return max(0, self.max_turns - self.turns_used)
 
    def spend(self, tokens: int) -> None:
        """Record one turn. Raises when either cap is exhausted."""
        self.turns_used += 1
        self.tokens_used += tokens
        if self.turns_used >= self.max_turns:
            raise BudgetExhausted(f"turn cap {self.max_turns} reached")
        if self.tokens_used >= self.max_tokens:
            raise BudgetExhausted(f"token cap {self.max_tokens} reached")
 
    def wrap_up_signal(self) -> str | None:
        """Tell the model to land the plane while it still has room to."""
        if self.remaining_turns <= 2:
            return "Two turns left. Summarise state and stop."
        return None
 
 
class BudgetExhausted(RuntimeError):
    """Raised when a run hits a cap. Never caught silently."""

Errors are the teaching signal, so write them like documentation

An agent reads your error messages more carefully than your users ever will, and unlike your users it cannot go ask someone. Anthropic's guidance is that error messages should communicate specific, actionable improvements and include correctly-formatted examples. A tool that returns ValidationError: invalid input teaches nothing. A tool that returns the failing field, the expected shape, and a working example changes the next attempt.

Two smaller details from the same guidance changed my tool design. Semantic, human-readable identifiers beat UUIDs, and the claim is specific: they significantly improve Claude's precision in retrieval tasks by reducing hallucinations. A model can pattern-match invoice_2026_04_acme in a way it simply cannot do with a hex blob. And tool namespacing has non-trivial effects depending on whether you prefix or suffix, which sounds like superstition until you watch a model consistently reach for the wrong one of two similarly named tools.

The detector I wrote, which you should treat as an opinion

Here's where I have to be honest about the state of the evidence. I needed a no-progress detector, because a run that spins burns real money and produces nothing anyone can use, and I needed it that week. I built one. It hashes the observable state after each turn, tracks repeated tool-call n-grams, and trips when the last several turns are structurally identical to earlier ones.

It works on the traces I had, and I shipped it because the alternative was letting runs spin. It is also entirely my own engineering judgment, and I could not find a primary source publishing a specific, validated no-progress or oscillation detection algorithm. Not state-hash repetition, not action edit distance, not n-gram repetition of tool calls. So I'm not going to dress mine up as established practice or put a number on how well it works.

What is documented, and what I'd build on before building a detector, are turn caps, RemainingSteps, and an external checklist that makes progress observable from outside the model's own account of itself. A detector guesses at progress. A checklist measures it.

Long-horizon work needs a harness, not a bigger window

For work that spans multiple context windows, the scaffolding around the model is called a harness, and the important finding is that compaction alone is not sufficient. Summarising well still leaves you with an agent that has no durable record of what it committed to.

Two failure modes show up by name. Over-ambition: agents try to one-shot the whole app, exhaust context mid-implementation, and leave behind half-finished features that nothing documents. Premature completion: a later instance, reading a summary rather than the work, declares the job done.

Both got fixed by the same thing, and it's mundane. A structured external checklist, held as a JSON feature list with more than 200 structured end-to-end test cases, each marked passes: false until proven otherwise, plus an explicit instruction that removing or editing tests is unacceptable. That last clause is not paranoia. An agent optimising for a green checklist will absolutely delete the failing test if you don't forbid it.

# harness.py
import json
from pathlib import Path
 
CHECKLIST = Path("features.json")
 
 
def load_checklist() -> list[dict]:
    """Read the durable feature list that survives every context window."""
    return json.loads(CHECKLIST.read_text())
 
 
def next_unfinished(checklist: list[dict]) -> dict | None:
    """Pick the next unproven feature, in declaration order.
 
    Declaration order, not model preference. Letting the agent choose its
    own next task is how you get four half-built features instead of one
    finished one.
    """
    for feature in checklist:
        if not feature["passes"]:
            return feature
    return None
 
 
def mark_pass(checklist: list[dict], feature_id: str, evidence: str) -> list[dict]:
    """Flip a feature to passing, but only with attached evidence.
 
    Evidence means a browser transcript, not a claim. Code inspection is
    explicitly not acceptable proof that a feature works.
    """
    if not evidence.strip():
        raise ValueError(f"{feature_id}: cannot pass without evidence")
    for feature in checklist:
        if feature["id"] == feature_id:
            feature["passes"] = True
            feature["evidence"] = evidence
    CHECKLIST.write_text(json.dumps(checklist, indent=2))
    return checklist

The verification detail from that same work is the one I'd tattoo somewhere. Claude initially marked features complete on the basis of code inspection alone. It read the code, the code looked right, the box got ticked. The fix was an explicit instruction to drive a browser and test the feature the way a human user would. "I read the implementation and it looks correct" is not evidence, from an agent or from anyone else.

If you want a map of how these things fail rather than a list of anecdotes, MAST is the one to read. It's a taxonomy from Berkeley of 14 failure modes across 3 categories, covering system design, inter-agent misalignment, and task verification. It was built from more than 1,600 annotated traces across 7 frameworks, and the human annotators reached a Cohen's kappa of 0.88, which is a level of agreement that makes the categories worth taking seriously rather than treating as one team's vocabulary.

What I got wrong first

I spent two months tuning prompts and about two weeks building loop machinery, and the two weeks produced almost all of the reliability. That ratio was backwards, and I'd have caught it before I made any promises if I'd measured pass^k instead of running the happy path once the night before a demo and feeling good about it.

The other thing I got wrong: I raised max_turns whenever a run died at the cap. Every single time, it felt like the right move, and every single time the extra turns went into the same spin. More turns is not better. Without a progress signal, raising the cap converts a failure into an expensive failure, and you pay for the privilege of watching it fail later.

The concrete next step, if you take one thing from this: pick your three most important tasks, run each one eight times against your current agent, and count how many succeed all eight times. That number is your real pass rate. Everything in this post is a response to what that number looks like the first time you compute it.