a-constrained-agent-computer-interface-beats-raw-shell-accessmechanismsingle paperpending review
Giving a coding agent a raw shell is worse than giving it a designed interface: SWE-agent's constrained Agent-Computer Interface — a file viewer windowed to 100 lines, a search tool that truncates large output, and an editor that lints before a change persists — roughly doubled SWE-bench resolution over raw shell access with the same model, because verbose output and unmanaged state are where the model loses its place.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Using the tools it is given · Agentic, Coding agent
Observed on
2024, GPT-4-class models on SWE-bench.
Sources
- The paper's central result and the origin of the term agent-computer interface. The comparison holds the model fixed and varies the interface.
- The same effect two years later from one narrow change: tagging each line with a content hash so edits cannot silently hit the wrong line improved nearly all of 16 models, Grok Code Fast 1 from 6.7% to 68.3%.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Giving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.Following an unfamiliar procedure · contested · unreviewed
- A large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.Whether the measurement made the finding · unreviewed
- When a model gets a real environment signal after acting — a test result, a tool error, a task-success indicator — reflecting on that signal in words and retrying substantially improves success rates on multi-step coding and decision-making tasks over a single attempt.Fixing its own mistakes
- For long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.Following instructions hidden in data · unreviewed
- When an agent must work over a very large tool catalogue, wrapping each tool in a natural-language interface that resolves the API schema internally — so raw schemas are retrieved on demand rather than enumerated in the model's prompt — contributes more to multi-step tool- call success than the surrounding planner, router, or verifier stages.Using the tools it is given · unreviewed
Notes
The SWE-agent brief carries one flagged figure, 2,294, which is the size of full SWE-bench and does not appear in the paper's text as fetched; the claim above does not use it.