Loop Engineering

An agent succeeds during a local test demonstration. A week later, it fails on the exact same task in a staging environment. Nothing about the prompt changed.

Prompt tuning optimizes the mean. What kills you in production is the variance, and variance is a property of the execution loop, not the prompt.

Chapter 0: Anatomy of an agent loop

An agent loop runs four stages: gather context, take action, verify work, repeat.

Most homegrown implementations skip verification. Without an evaluation gate the loop keeps taking actions and never notices that an intermediate output has drifted off its constraints.

Step 1: Measuring pass^k instead of pass@1

Shunyu Yao and co-authors' τ-bench, published in June 2024, measures tool-calling reliability using pass^k: the percentage of tasks where all k independent attempts succeed consecutively.

Their headline result is the gap between the two metrics. State-of-the-art function-calling agents (gpt-4o at the time of writing) succeed on under 50% of tasks on a single attempt, and pass^8 falls below 25% in the retail domain.

Measuring single-run success gives you an optimistic number. Running the same task k times and reporting pass^k is what exposes how stable the loop actually is.

The tell: if you have never run the same task twice under the same conditions, you do not know your failure rate. You know your best case.

Step 2: The verification ladder: deterministic checks first

Verification should follow a tiered cost hierarchy:

1. Deterministic syntax and type checks: fast, local linters (such as tsc, ruff, or AST parsers).
2. Programmatic assertions: test suites and schema validators.
3. Visual evaluation: rendering screenshots to verify UI state.
4. Model-based grading: LLM-as-a-judge for semantic nuances that code cannot test.

# loop.py
from dataclasses import dataclass, field
from typing import Callable, Literal, List, Tuple
 
Verdict = Literal["pass", "fail", "unknown"]
 
@dataclass
class Check:
    """A single verification check with associated execution cost."""
    name: str
    cost: int
    run: Callable[[str], Verdict]
 
@dataclass
class VerificationLadder:
    """Executes checks in ascending cost order, halting on first failure."""
    checks: List[Check] = field(default_factory=list)
 
    def verify(self, artifact: str) -> Tuple[Verdict, List[str]]:
        executed_checks: List[str] = []
        for check in sorted(self.checks, key=lambda c: c.cost):
            executed_checks.append(check.name)
            verdict = check.run(artifact)
            if verdict == "fail":
                return "fail", executed_checks
        return "pass", executed_checks

Ordering checks by cost means broken syntax fails on the cheapest rung, before you pay a model judge to read it.

The tell: if your linter and your judge run on the same artifact in the same turn, the ladder is not sorted.

Step 3: Managing context budgets and compaction

Chroma's context rot report from July 2025 evaluated 18 models — GPT-4.1, Claude 4, Gemini 2.5, Qwen3 — and found that models do not use their context uniformly: performance grows increasingly unreliable as input length grows, on tasks as simple as retrieval.

Models degrade when the window fills with unfocused tokens. Stale tool output competes with your instructions for attention.

Manage context through active compaction:

• Blank or truncate stale tool observations once downstream actions consume their outputs.
• Summarize multi-turn execution histories into concise decision logs when approaching window limits.
• Delegate verbose exploratory tasks to sub-agents that return concise summary payloads to the parent orchestrator.

# compaction.py
from dataclasses import dataclass
from typing import List, Dict, Any
 
@dataclass
class Transcript:
    messages: List[Dict[str, Any]]
    files_touched: List[str]
 
def clear_stale_tool_outputs(messages: List[Dict[str, Any]], retain_latest: int = 2) -> List[Dict[str, Any]]:
    """Replaces older verbose tool outputs with truncated placeholders."""
    seen_tool_messages = 0
    updated_messages = []
 
    for msg in reversed(messages):
        if msg.get("role") == "tool":
            seen_tool_messages += 1
            if seen_tool_messages > retain_latest:
                msg = {**msg, "content": "[output cleared from working context]"}
        updated_messages.append(msg)
 
    return list(reversed(updated_messages))

Step 4: Enforce execution bounds and token caps

An unbounded loop will cycle forever on an API error it does not know how to interpret. Enforce explicit turn and token budgets:

# budget.py
from dataclasses import dataclass
 
class BudgetExhaustedError(RuntimeError):
    """Raised when turn or token limits are exceeded."""
    pass
 
@dataclass
class ExecutionBudget:
    max_turns: int
    max_tokens: int
    turns_used: int = 0
    tokens_used: int = 0
 
    @property
    def remaining_turns(self) -> int:
        return max(0, self.max_turns - self.turns_used)
 
    def record_step(self, token_count: int) -> None:
        self.turns_used += 1
        self.tokens_used += token_count
        if self.turns_used >= self.max_turns:
            raise BudgetExhaustedError(f"Turn limit ({self.max_turns}) exceeded.")
        if self.tokens_used >= self.max_tokens:
            raise BudgetExhaustedError(f"Token budget ({self.max_tokens}) exceeded.")
 
    def should_terminate(self) -> bool:
        return self.remaining_turns <= 1

Step 5: Anchor progress to external state checklists

Long-running workflows spanning multiple steps require externalized state tracking. Models can declare tasks complete prematurely if progress is tracked only within ephemeral prompt contexts.

Maintain an external checklist in structured JSON to track task completion:

# checklist.py
import json
from pathlib import Path
from typing import Any, Dict, List, Optional
 
CHECKLIST_PATH = Path("execution_checklist.json")
 
def load_checklist() -> List[Dict[str, Any]]:
    return json.loads(CHECKLIST_PATH.read_text())
 
def get_next_pending_task(checklist: List[Dict[str, Any]]) -> Optional[Dict[str, Any]]:
    for item in checklist:
        if not item.get("passed"):
            return item
    return None
 
def record_task_verification(task_id: str, test_evidence: str) -> None:
    if not test_evidence.strip():
        raise ValueError(f"Task {task_id} requires concrete test execution evidence.")
    
    checklist = load_checklist()
    for item in checklist:
        if item["id"] == task_id:
            item["passed"] = True
            item["evidence"] = test_evidence
            break
            
    CHECKLIST_PATH.write_text(json.dumps(checklist, indent=2))

Requiring explicit execution output (such as passing test runners or status codes) prevents models from assuming completion without verification.

The tell: if a task can be marked done without a command output pasted next to it, the checklist is decoration.

When this is the wrong choice

• The task runs once and a person reads the result. Everything above buys reliability across repeated runs. If nobody reruns the task, pass^k and pass@1 are the same number, and the loop machinery costs turns for nothing.
• You have no cheap deterministic check to put on the bottom rung. The ladder works because tsc or ruff costs almost nothing to run every turn. Where the output is prose with no compiler behind it, every rung is a model call and verification costs about as much as generation.
• The whole job fits in one context window. Compaction and external checklists exist to survive window pressure over a long horizon. Below that, clearing stale tool output deletes information the model still needs, and the JSON checklist is a second source of truth to keep in sync.

Engineering checklist for reliable loops

1. Benchmark pass^k: measure stability across repeated runs rather than optimizing single-turn demos.
2. Implement tiered verification: evaluate cheap, deterministic type and syntax checks before calling model judges.
3. Prune context aggressively: clear stale tool observation payloads to prevent context degradation.
4. Enforce hard execution bounds: configure explicit turn and token ceilings.
5. Externalize task checklists: track state transitions and verification evidence outside ephemeral model context.