a-fixed-model-can-improve-its-own-harness-with-a-regression-gated-loopWith weights frozen and no stronger model in the loop, an agent can raise its own pass rate by rewriting the scaffolding it runs inside — clustering its failures, proposing minimal edits to prompts, tools and policy, and keeping only edits that improve one split without degrading another: all 9 of 9 model-by-benchmark pairs improved, by up to 40.6 percentage points (GLM-5 on AppWorld, 44.4% to 85.0%), with weak models gaining most; but the promotion gate reads the held-out split, so the reported held-out gain is not clean generalisation, and there is no matched-compute baseline.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Fixing its own mistakes · Agentic, Coding agent
Observed on
2026, MiniMax M2.5 / Qwen3.5-35B-A3B / GLM-5 on Terminal-Bench 2.0, SWE-bench Verified, AppWorld.
Sources
- Model, tools, budget, evaluator and splits held fixed, so the harness is isolated as the cause. Largest relative gain 132%, weakest 52.0% to 55.5%. Two repeated attempts per score; no comparison to a human-engineered harness; edits may be benchmark-specific. The authors state the held-out caveat themselves.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- The large reported gains from fine-tuning on tool-call trajectories come from small open models starting far behind — distilling a few hundred trajectories from a stronger model moves them a long way. Whether the same procedure helps a model that is already good at tool use is a separate question this evidence does not answer.Using the tools it is given
- In open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.Fixing its own mistakes · unreviewed
- A large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.Whether the measurement made the finding · unreviewed
- Reporting how much a lab's models are used in its own research — tokens, lines of code, inference compute, experiments per researcher — cannot show that the models are accelerating the research, because every one of those is an input. Establishing the loop needs an outcome variable over time: algorithmic efficiency as a function of capability, which is the edge that turns a pipeline into a feedback loop.Whether the measurement made the finding
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · unreviewed
Notes
The closest existing implementation of this project's own first ambition, and its stated weakness is the one docs/ambitions.md names: a loop that grades its own edits on the split it will be scored on has closed the circle. The fix it lacks — a held-out set the loop never sees and cannot grow — is the rule already in force for the backtest here.