fact-verification · activeChecking claims against evidence
Models accept confident false premises and resolve conflicts between sources by majority or recency rather than by reliability.
Also called: epistemic verification, truth adjudication, knowledge conflict
Tags: chat-assistant, data-analysis, knowledge, rag-qa
Given a claim and some evidence, a capable model decides whether the claim is supported, notices when the evidence conflicts with what it believes, and weighs sources by reliability rather than by count, order, or tone.
Claims
- mechanismsingle paperGiven several conflicting but equally legitimate passages, a model's answer is driven disproportionately by whichever passage comes first, and the skew sits in the representation of the combined prompt rather than in decoding — which is why reordering the passages does not remove it and why steering the activations does.
- observationsingle paperWhen an LLM agent must reconcile user-supplied identity credentials against database records before a sensitive read or write, frontier and open models frequently skip the cross-field consistency check and act anyway, and the failure rate barely moves whether the request is simple or has several parallel sub-requests and whether the forged field is visually near-identical to the true one or completely unrelated.
- mechanismsingle paperModels endorse widely held falsehoods, showing weak verification against what they know.
- mechanismsingle paperModels readily adopt a single counter-memory passage when it is coherent, and when sources conflict they follow the majority and show confirmation bias toward their own beliefs.
- mechanismsingle paperTraining the model to decide when to retrieve and to critique whether passages support its output improves factuality and citation accuracy.
- mechanismsingle paperWhen training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.
- observationsingle paperWhen VLM agents must verify claims against interactive visualizations where no claim is answerable from the initial viewport, giving them an interaction budget of ten actions does not reliably beat answering immediately from the first screenshot — some models score slightly lower with interaction — because partial, unplanned exploration leaves the model with enough evidence to abandon its prior but not enough to replace it.
Techniques
- Check long answers fact by facttoolingSplit a long answer into atomic claims and verify each against search or a corpus.
- Ground answers in retrieved documentsretrievalRetrieve relevant documents and generate from them, with the model deciding when retrieval is needed.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.