A personal study of what LLMs are actually good and bad at
Not a scoreboard — a set of directional, scoped claims, each tied to the sources that support or contest it. Capabilities are topics; claims are the actual findings.
Incomplete, and partly unreviewed — 82 of 167 claims are drafted from a paper and not yet checked by a person, and are marked as such. Nothing is presented with more confidence than it has. How this is collected.
Worth a look
2026-09-11 · forwarded by Russ: a vocabulary for the harness around a model, and a ratchet that maps onto this catalog's own loopA nine-page playbook presenting production agent design as six layers around the model: guides (feedforward instructions), sensors (feedback checks), a bounded agentic loop, persistent memory, permissions and budgets, and observability. Its organising idea is the ratchet: every failure becomes a permanent fix in the harness rather than a patch to a prompt. It compiles harness-only performance gains from five secondhand sources and supplies a vocabulary — guides versus sensors, computational versus inferential sensors, trip wires, capability budgets, escalation packets — that maps unusually well onto this catalog's own machinery.
Claims drawn from it
- Given several conflicting but equally legitimate passages, a model's answer is driven disproportionately by whichever passage comes first, and the skew sits in the representation of the combined prompt rather than in decoding — which is why reordering the passages does not remove it and why steering the activations does.
- contested Where a passage sits in a long input changes how much the model uses it — accuracy is highest when the needed information is at the very start or the very end and lowest when it is in the middle — so ordering retrieved passages to put the most relevant ones first is a real lever on accuracy.
the source in full · featured as something worth reading, not as something judged correct
Most recently checked claims
- mechanismsingle paperchecked 2026-09-11Giving a coding agent a raw shell is worse than giving it a designed interface: SWE-agent's constrained Agent-Computer Interface — a file viewer windowed to 100 lines, a search tool that truncates large output, and an editor that lints before a change persists — roughly doubled SWE-bench resolution over raw shell access with the same model, because verbose output and unmanaged state are where the model loses its place.
- observationsingle paperchecked 2026-09-11With weights frozen and no stronger model in the loop, an agent can raise its own pass rate by rewriting the scaffolding it runs inside — clustering its failures, proposing minimal edits to prompts, tools and policy, and keeping only edits that improve one split without degrading another: all 9 of 9 model-by-benchmark pairs improved, by up to 40.6 percentage points (GLM-5 on AppWorld, 44.4% to 85.0%), with weak models gaining most; but the promotion gate reads the held-out split, so the reported held-out gain is not clean generalisation, and there is no matched-compute baseline.
- mechanismmechanism reasoningchecked 2026-09-11In an agent's verify-and-fix loop, deterministic checks — tests, linters, schema validators — should be exhausted before any model-based judge is added, because they are free, fast and give the same verdict every time, while an LLM-as-judge costs tokens per run, returns non-deterministic verdicts, and so belongs only on properties no rule can express, and even then as an advisory signal until measured against human review.
- mechanismmechanism reasoningchecked 2026-09-11A correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.
- mechanismsingle paperchecked 2026-09-11A large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.
- observationsingle papercontestedchecked 2026-09-11Giving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.
Reasoning
Arithmetic, logic, planning, and compositional generalization.
- Digit-level arithmetic
Multi-digit arithmetic, counting, and symbol manipulation fail in ways tied to tokenization and memorization.
- Generating and editing working code
Whether a model can write correct code from a specification, and go beyond isolated functions to real repository-scale editing and debugging.
- Predicting future events
Whether a model can make well-calibrated predictions about events that haven't happened yet, not just recall or verify existing facts.
- Reasoning about time in video
Tracking motion, ordering, causality, and change across video frames rather than treating video as isolated stills.
- Turning plans into physical action
Closing the loop from perception to action in a body or simulator, where mistakes have physical consequences and cannot be retried freely.
- Understanding speech and audio
Hearing what is said and how it is said — tone, emotion, speaker identity, and events in a soundscape.
Knowledge
Factual recall, verification, long-tail and time-sensitive facts.
- Checking claims against evidence
Models accept confident false premises and resolve conflicts between sources by majority or recency rather than by reliability.
- Grounding answers in the image
Saying only what the image supports, rather than describing plausible objects that are not there.
- Not generalizing "A is B" to "B is A"
A model that learned a fact in one direction often cannot recall it when asked in the reverse direction.
- Predicting future events
Whether a model can make well-calibrated predictions about events that haven't happened yet, not just recall or verify existing facts.
- Stating false facts confidently
Models generate plausible but false statements, citations, and details, with no signal that they are unsure.
- Understanding speech and audio
Hearing what is said and how it is said — tone, emotion, speaker identity, and events in a soundscape.
Context and memory
Using long inputs, tracking state through a task, and remembering across sessions.
- Keeping its own context clean
Agents fill their working context with re-read files, restated notes, and excess documentation until quality degrades.
- Losing information in long inputs
Accuracy drops as inputs grow and is worst for information placed in the middle of the context.
- Reasoning about time in video
Tracking motion, ordering, causality, and change across video frames rather than treating video as isolated stills.
- Remembering across sessions
Assistants fail to recall and apply facts, preferences, and decisions from earlier sessions.
- Tracking state through a long task
Models lose track of the current state of entities, files, or an environment after many steps.
Behavior
Sycophancy, over-refusal, instruction following, calibration, and tone.
- Grounding answers in the image
Saying only what the image supports, rather than describing plausible objects that are not there.
- Saying what actually drove the answer
Whether a model's stated reasoning reflects the computation that produced its answer, rather than a plausible story told after the fact.
- Strategic deception and detecting it
Whether a model can deliberately induce false beliefs in others when a task calls for it, and detect deception from others.
- Telling the user what they want to hear
Models shift answers toward the user's stated view, cave to pushback, and mirror mistaken premises.
Agentic
Tool use, procedure following, self-repair, and handling conflicting goals.
- Fixing its own mistakes
Whether and how a model can diagnose and correct a wrong answer once it's already made.
- Following an unfamiliar procedure
Models drift from a new multi-step procedure or set of constraints back to familiar routines.
- Generating and editing working code
Whether a model can write correct code from a specification, and go beyond isolated functions to real repository-scale editing and debugging.
- Prioritizing safety under conflicting goals
When a task goal competes with a safety rule or a required check, models sometimes drop the check to finish the task.
- Turning plans into physical action
Closing the loop from perception to action in a body or simulator, where mistakes have physical consequences and cannot be retried freely.
- Using the tools it is given
Models often fail to reach for provided tools, skills, or documented procedures, or call them with wrong arguments.
Security
Prompt injection, insecure code, package hallucination, and data leakage.
- Following instructions hidden in data
Models treat instructions found in retrieved documents, tool outputs, or web pages as if they came from the user.
- Writing secure code and dependencies
Generated code carries known vulnerability patterns, and models suggest packages that do not exist.
Evaluation
Biases when models judge, benchmark contamination, and self-assessment.
- Biased when judging other outputs
A model used as a judge favors whichever answer is shown first, longer answers, and its own outputs.
- Saying what actually drove the answer
Whether a model's stated reasoning reflects the computation that produced its answer, rather than a plausible story told after the fact.
- Whether the measurement made the finding
Whether a reported result describes the model or the way it was scored — the metric, the prompt format, the contamination, the baseline.