evaluation-validity · activeWhether the measurement made the finding
Whether a reported result describes the model or the way it was scored — the metric, the prompt format, the contamination, the baseline.
Tags: evaluation
A finding can be an artifact of how it was measured. A discontinuous scoring rule turns smooth improvement into an apparent jump; a baseline given a smaller budget loses; a benchmark in the training data flatters everyone. This capability collects claims about when a measurement stops reflecting the thing it names. It sits oddly beside the others because it is not a property of a model, but every claim in this index rests on some measurement, so it is the one capability whose failures propagate everywhere.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
In scope when the paper's finding is about the measurement rather than the model: showing a result changes or disappears under a different metric, prompt format, baseline budget, or decontaminated split. NOT in scope: a benchmark paper reporting how models score, which belongs to whichever capability it tests; and not general "we need better evals" position papers with no demonstration.
Claims
- mechanismsingle paperWhen comparing a weight-shared recursive model against a depth-matched plain transformer on code generation, the ranking read from a single checkpoint can invert during training, because the larger-parameter arm passes its validation optimum and overfits while the parameter-lean recursive arm keeps improving — so a late-epoch "recursion fits better" reading is a degradation artifact, not a capability difference.
- observationsingle paperWhen grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.
- mechanismsingle paperA large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.
- mechanismmechanism reasoningReporting how much a lab's models are used in its own research — tokens, lines of code, inference compute, experiments per researcher — cannot show that the models are accelerating the research, because every one of those is an input. Establishing the loop needs an outcome variable over time: algorithmic efficiency as a function of capability, which is the edge that turns a pipeline into a feedback loop.
- observationsingle paperOn the best current calibration, AI is not yet accelerating its own development in a self-sustaining way: the modelled threshold is that a one-unit gain in model capability must buy at least 15% higher AI R&D productivity, and the back-of-envelope figure from reported engineer uplift since coding agents launched is about 9% — below it, but rising, so the gap is a current reading rather than a ceiling.
- mechanismsingle paperOn multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.
- mechanismsingle paperWhen research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.
- observationsingle paperOn span-level hallucination-detection leaderboards where absolute scores are low, bootstrap resampling of the test set moves top systems across wide rank intervals, so point-estimate ordering does not establish that one detector beats another.
- mechanismsingle paperIn synthetic hallucination benchmarks where faithful items are human-written and hallucinated items are LLM-rewrites, a detector's binary score partly measures human-vs-machine style rather than factuality: feeding faithful LLM rewrites of the same truthful articles makes some fine-tuned encoders flag nearly all of them as hallucinated.
- observationsingle paperFor training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.
Techniques
- Deterministic checks before LLM judgesprocessWire linters, test suites and schema validators into the agent's loop before adding any model-based judgment.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.