visual-grounding · activeGrounding answers in the image
Saying only what the image supports, rather than describing plausible objects that are not there.
Tags: knowledge, behavior
The vision-side analogue of hallucination, kept separate because the mechanisms and mitigations differ: text hallucination is addressed with retrieval and verification, while visual ungroundedness is about whether the model attends to image evidence at all, or leans on language priors about what usually appears in such a scene.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
In scope when the paper measures or mitigates a model asserting content the image does not support — object hallucination, invented attributes or relations, answers driven by language priors rather than pixels. NOT in scope: general recognition accuracy, and not text-only hallucination, which belongs to the hallucination capability.
Claims
- observationsingle paperIn 7-8B vision-language models, fine-tuning visual and text embeddings toward perturbation-averaged and ground-truth anchors before cross-modal contrastive alignment lowers object and attribute hallucination rates on captioning and existence benchmarks without reducing object coverage or general VQA accuracy.
- mechanismsingle paperIn 7B vision-language models that prune visual tokens during decoding for efficiency, hallucination rises as the generated caption gets longer, because pruning is irreversible and discards fine-grained evidence needed at later decoding steps; keeping the pruned tokens as a retrievable bank and pulling them back when next-token entropy spikes recovers the grounding without a second forward pass.
- mechanismsingle paperFor detecting ungrounded object mentions in open-ended captions from white-box vision-language models, a single internal support signal (token likelihood, attention share, or image-text similarity) confuses support that comes from object-specific image evidence with support that comes from scene plausibility or the generated text prefix; scoring both whether image tokens carry object-specific evidence and whether support is more image-derived than prefix-derived separates these and raises detection AUROC.
- observationsingle paperIn diffusion vision-language models generating free-form captions, object tokens that are hallucinated commit at later denoising steps and with lower per-token confidence than grounded object tokens, so the commit-step trajectory is an available internal hallucination signal that has no counterpart in autoregressive decoding.
- mechanismsingle paperIn LLaVA-1.5-7B captioning, restricting a LoRA adapter to attention heads picked by a hallucination diagnosis (attention-to-image drop around hallucinated object words, then ablation screening) reduces object hallucination, while an identically trained layer-matched random-head adapter does not — so the selection, not the added adapter capacity, carries the effect.
- mechanismsingle paperWhen a multimodal model is asked an object-existence question over a set of images at once, it is less accurate than when the same images are queried one at a time and the answers combined, so cross-image integration adds errors beyond the per-image perception errors.
- observationsingle paperIn open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.
- mechanismsingle paperFor short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.
- observationsingle paperFor training-free inference-time hallucination interventions in 7B-class vision-language models, increasing intervention strength (finer regional evidence partitioning, longer attention-routing windows, stronger textual perturbation) keeps lowering CHAIR object-hallucination scores while producing empty or reduced-coverage responses, so lower hallucination numbers at high intervention strength partly reflect the model saying less rather than grounding better.
Techniques
None yet.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.