code-generation · activeGenerating and editing working code
Whether a model can write correct code from a specification, and go beyond isolated functions to real repository-scale editing and debugging.
Tags: reasoning, agentic
Distinct from secure-coding, which is specifically about vulnerability patterns and dependency hallucination. This is about raw functional correctness — does the generated code actually pass held-out tests — and, at the harder end, whether a model can navigate and edit an existing codebase rather than writing an isolated function from scratch. Static, fixed-problem benchmarks saturate and leak into training data over time, which is why contamination-resistant, continuously-updated benchmarks matter here more than in most areas.
Claims
- mechanismsingle paperIn agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.
- observationsingle paperWhen a code-generation prompt combines a prompt-enforced output format (JSON or XML), a persona, and urgency framing at once, pass@1 on function-level Python problems can fall well below the sum of each constraint's individual effect, even when each constraint alone is neutral or helpful — observed in the GPT-4o family and absent in the GPT-4.1 family and o3-mini.
- observationsingle paperFor data-science Python tasks where several correct solutions differ by orders of magnitude in runtime, a model's correctness rate does not predict the efficiency of the code it writes: rankings invert between the two, code-specialized open models are less efficient than general open models of similar size class, and the fastest model rotates by library (NumPy vs Pandas vs PyTorch vs Polars).
- mechanismsingle paperGrading generated code by whether it actually passes held-out functional tests, rather than by surface similarity to a reference solution, is what makes code generation gradeable at scale — and it's how the field has graded it since the earliest large benchmarks.
- observationsingle paperOn repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.
- mechanismsingle paperStatic, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.
- mechanismsingle paperIn UI-to-code generation, where a rendered screenshot gives genuine external feedback, iterative self-critique still drifts below the initial draft over ten rounds, because a code edit changes the rendering non-locally and a fix for one region breaks regions that were already faithful; restricting each round to one typed, scoped repair target and carrying past targets forward as an avoid-list reduces that drift.
Techniques
None yet.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.