A failed test should change the next step
Imagine an agent fixing a date filter in a web application. The brief says that selecting October 12 must include records from that entire day. Its first patch excludes records after midnight. If the agent simply rereads its own explanation, it may confidently report success while the bug survives.
Give it a test runner, and the next decision can use something firmer: an afternoon record is missing. The agent inspects the boundary calculation, proposes a different patch and runs the check again. That sequence is an execution loop: take an action, observe its result, decide what follows. Anthropic's foundational agent guidance describes this use of feedback from the environment. [L1]
For this hypothetical task, define success before editing: the selected date includes its expected records, adjacent dates stay excluded and existing tests keep passing. Otherwise, the agent can spend several attempts solving the wrong problem.
The useful part of a retry is the new evidence that changes the next attempt.
Editorial highlight · Original editorial line, not a third-party quotation.
Give the next attempt a usable record
The loop needs state: the requested behavior, current code revision, files changed, exact failing check, attempts remaining and unresolved questions. A later session also needs to know which results belong to which revision. A stale green test beside a newer patch is misleading evidence.
Memory is the retained information made available to later decisions. In this example, a short handoff could preserve the failed midnight assumption and link to the test output. Saving that record does not update the model's weights. It changes the information the next run can use.
Keep the user's requirements separate from the agent's working notes. Anthropic's recent automation example separates fixed preferences from mutable state; it also warns that untrusted content can contaminate memory. [L4]
A graph sets the routes
Draw this process as a graph and each node represents a step: inspect, edit, test or request review. The connecting edges specify possible routes. A failed test can lead back to editing; an unclear requirement can lead to a person; exhausted retries can lead to a stopped run.
The graph makes these transitions explicit. It does not, by itself, teach the agent a better repair strategy. Repeating a route can reproduce the same mistake. Improvement requires useful feedback, a changed decision and a check that the change helped. Prompts and tool instructions still shape those decisions.
Separate the work from the verdict
For the date-filter example, keep acceptance tests outside the editing agent's write access. A separate runner checks the proposed revision and records what actually happened. Before editing, ask a person to settle ambiguous requirements, such as which time zone defines the selected date.
A model can assist that review, but a second agent can share the first agent's blind spots. Anthropic's harness research found value in separating generation from evaluation while explicitly acknowledging that evaluators can still grade too generously. [L2]
Set a concrete boundary: one initial attempt and at most two retries, within a fixed time and spending budget. Stop earlier if the requirement is unclear. At the limit, return the patch, evidence and unresolved failure. In this example, a person approves merging or deployment, even if the checks pass.
Measure improvement beyond one run
A repaired example shows only that example. To judge the system, compare it with a simpler baseline on representative tasks, including cases kept out of the tuning process. Anthropic's evaluation guidance recommends held-out cases and controlled changes to expose overfitting and regressions. [L3]
Track successful completion, false declarations of success, human intervention and cost per successful task, including failed attempts. Check slow runs as well as average time. Keep the test conditions comparable and retain failure traces. A loop earns more autonomy when that evidence supports it; extra iterations alone prove nothing.
Sources and references
- Building effective agents
- Harness design for long-running application development
- Automating eval design and hillclimbing with Claude
- Building effective agent automations
- Lamis Mukta — Learning while you sleep: Beyond memory to dreaming — AI Native DevCon June 2026
Got a different take? I’d love to hear it. hello@maisfps.com
