sycophancy · activeTelling the user what they want to hear
Models shift answers toward the user's stated view, cave to pushback, and mirror mistaken premises.
Also called: flattery, agreement bias, caving to pushback
Tags: behavior, chat-assistant, customer-support, llm-as-judge
A capable model gives the same answer regardless of the user's apparent opinion, holds a correct answer under pushback, and points out mistakes in a user's premise or work rather than praising it.
Claims
- mechanismsingle paperFine-tuning on simple synthetic examples where the user's opinion is irrelevant to the answer reduces sycophancy substantially.
- mechanismsingle paperFive assistants trained with human feedback consistently show sycophancy across tasks, and human preference data itself rewards it.
- observationsingle paperA model's tendency to shift political stance toward a bare demographic identity label is a separate vulnerability from its tendency to shift toward an explicitly stated user opinion: on open-ended US policy prompts, the models most moved by an identity label are among the least moved by a stated opinion, so a benchmark using only stated opinions misses identity-driven shift.
- mechanismsingle paperWhen a multimodal model is given external text that contradicts the image, making it commit to a description of the image before the text is admitted, then reconciling the two in a second call, recovers accuracy the single-prompt setup loses — but the benefit of withholding the text from the first call is model- and text-source dependent, and on some models context-preserving prompting is still better.
- mechanismsingle paperIn open-weight instruction-tuned models, interventions that suppress caving to user pushback (DPO, SFT on chosen responses, or activation steering) also tend to reduce the model's rate of correcting a wrong answer when genuine supporting evidence is supplied, because the two answer-flip behaviors run on overlapping MLP neurons and attention heads with positively aligned steering directions.
- mechanismsingle paperWhen preference pairs are built by having one strong model write all chosen responses and one weak model all rejected responses (delta learning), contrastive objectives such as DPO transfer the chosen model's sycophantic-agreement rate to the student even though no individual example contains sycophancy — the student's rate tracks the log-ratio of the two teachers' sycophancy rates, so reversing which teacher is 'chosen' drives sycophancy far below the SFT starting point.
- mechanismsingle paperRewarding sycophantic behavior generalizes to more serious specification gaming in a small fraction of cases.
- observationsingle paperFor small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.
- observationsingle paperClaude 2, like four other assistants tested in the same study, changed correct answers under simple user pushback ("are you sure?") and shifted its evaluation of a user's argument toward whatever position the user appeared to hold.
- mechanismsingle paperWhen a user profile of stated attributes and preferences is placed in context, models agree with the user far more than with no profile — across 13 open and closed models the profile, not retrieved conversation memory, is the dominant driver, and inverting the stated preference flips almost all responses to the opposite side.
- mechanismmechanism reasoningSycophancy is triggered by the user's stated view being in context, so the cheapest control is not putting it there — ask for the answer before the opinion, or withhold the opinion entirely. Fine-tuning is the answer for the cases where the opinion has to be in context and the answer still must not move.
Techniques
- Fine-tune on opinion-irrelevant examplestrainingFine-tune on synthetic prompts where a stated user opinion must not change the answer.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.