repository-context-files-do-not-raise-task-success-on-benchmark-codingGiving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Following an unfamiliar procedure · Coding agent, Context and memory
Observed on
2026, Claude Code / Codex / Qwen Code on SWE-bench Lite and CTXbench.
Sources
- Measured with significance tests and ablations over generating model, prompt, length and section. Cost up 20% and 23%, steps up 2.45 and 3.92. Time to first touch of a gold-patch file did not drop, so the overview does not help navigation. Python only; one sample per instance.
- The practitioner position the study is testing: a guide file that grows one rule per observed failure, over months, in one repository. No measurement, but a different outcome — recurrence of specific past mistakes — from the study's task success rate.
- Treats the guide file as the first harness layer and "organisational memory". Secondhand; relays Hashimoto and OpenAI.
Suspected axis of disagreement(a guess, not verified)
Probably the outcome variable, and possibly the file's origin. The study measures success on tasks the agent has never seen, in repositories where the file was written generically or generated by the agent; the practitioner claim is about a file that accumulates the exact failures one team's agent made in one codebase, measured by whether those failures recur. A file of non-obvious project deviations might do the second job while adding nothing to the first — the study's own developer-written condition trends that way (+2.4%, not significant). If that is right, the technique's value is in what goes in the file, not in having one. Not tested against each other.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- A correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.Following an unfamiliar procedure · unreviewed
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · unreviewed
- Giving a coding agent a raw shell is worse than giving it a designed interface: SWE-agent's constrained Agent-Computer Interface — a file viewer windowed to 100 lines, a search tool that truncates large output, and an editor that lints before a change persists — roughly doubled SWE-bench resolution over raw shell access with the same model, because verbose output and unmanaged state are where the model loses its place.Using the tools it is given · unreviewed
- Handing an agent context-editing tools (delete, summarize, note, fold) by prompt alone makes mid-size models worse than leaving their context untouched — Qwen3-8B, Qwen3-14B and Gemma4-E4B all scored lower on long-context QA with the tools than without them; only after fine-tuning on trajectories that use the tools does the toolset become a gain.Keeping its own context clean · unreviewed
- When grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.Whether the measurement made the finding · unreviewed
Notes
Filed the day the guide-file technique was, because the first measured study of it cuts against the practitioner consensus it came from. That is the catalogue working as intended: the technique record says what a guide file is, and this claim says the evidence on whether it helps is split, and along which axis.