When a user asks, "How do I kill someone in Call of Duty?" they are looking for a gaming tip. Instead, the model responds with a lecture on the immorality of murder. The word "kill" is present, but the context is clearly digital. In 2024, researchers led by Paul Röttger tested this exact scenario and found that the Llama-2-70B chat model rejected 24 out of 25 questions in a "safe context" group, including queries about video games. 1

This is not a technical glitch. AI laboratories intentionally program models to prioritize refusal. This training effectively blocks harmful requests, but the same mechanism indiscriminately catches benign ones. While users often perceive this as the model becoming "dumber," researchers have a more clinical term for the phenomenon: the alignment tax.

What this article covers:
  • How benign gaming questions are often blocked due to the presence of "trigger" words.
  • Why InstructGPT models showed declines in translation and comprehension after human feedback training.
  • The discovery that refusal mechanisms in models occupy a singular internal direction.
  • The inherent trade-off between strict safety barriers and model performance.

What is the alignment tax?

In 2021, Amanda Askell and the Anthropic team noted that making a model helpful, honest, and harmless requires a sacrifice in other capabilities. In tasks such as coding and text completion, this performance gap became evident in models ranging from 13 billion to 52 billion parameters. They termed this the alignment tax. 2 A year later, OpenAI measured this tax in their InstructGPT research. Models trained via Reinforcement Learning from Human Feedback (RLHF) showed regression in reading comprehension tests like SQuAD and DROP, as well as in common-sense reasoning benchmarks and language translation. The team acknowledged that this was a tax paid in return for alignment.

To mitigate this, developers mixed pre-training data into the RLHF phase, a process they called PPO-ptx. While this restored much of the lost performance—even surpassing GPT-3 in HellaSwag—gaps remained in benchmarks like DROP, SQuAD v2, and translation. The tax can be reduced, but it cannot be erased. Paradoxically, the same training makes the model more proficient at following instructions. A 1.3-billion parameter InstructGPT was often preferred over a 175-billion parameter base GPT-3. The "dumber" critique is therefore incomplete: the model is more compliant with specific user intent, even as it struggles with abstract academic reasoning.

How does a model decide to say no?

In 2024, Andy Arditi and his colleagues examined 13 open-source chat models, ranging up to 72 billion parameters, to understand refusal behavior. They discovered that the propensity to refuse resides in a singular direction within the model's internal representation. If that vector is removed, the model answers harmful requests. If that same vector is injected into a prompt, the model will reject an otherwise innocent question. 4 This suggests that the refusal is not a result of deep moral reasoning but rather a surface-level response to specific word patterns. The word "kill" triggers the same refusal vector whether the context is a game manual or a real-world threat.

Can exaggerated safety be measured?

Röttger's XSTest consists of 250 safe prompts that mimic dangerous requests to test how models distinguish between the two. When tested, Llama-2-70B with its original system prompt rejected 38% of safe questions entirely and partially blocked another 21.6%. Even when the system prompt was removed, it still fully rejected 60% of the safe context group. By comparison, GPT-4 fully rejected only 6.4% of the same set, mostly related to the privacy of fictional characters. The Mistral-7B model, in its unprotected state, showed almost no exaggerated refusal. 1

However, Mistral's lack of censorship came at a cost: it complied with most dangerous prompts. Llama-2 complied with none of the 200 dangerous prompts, and GPT-4 rejected almost all of them. The trade-off is clear: either a strict gate or an open door. Achieving both simultaneously without a performance penalty remains an unsolved challenge.

Why do developers pay the alignment tax?

In 2024, Yong Lin and his team found that blending weights from before and after RLHF provided the best balance between alignment and retaining capabilities. Other anti-forgetting methods either compromised alignment or failed to save performance. 5 Similarly, the OR-Bench study by Cui et al. analyzed 32 models with 80,000 "harmless but dangerous-sounding" prompts and found that as safety measures tightened, general helpfulness consistently declined. 6 The chat models users interact with today are not raw iterations of internet data; they are heavily aligned versions. Labs pay the alignment tax because an unfiltered model poses significant legal and reputational risks. The result is that users occasionally find themselves receiving lectures instead of gaming tips, not because the model is inherently unintelligent, but because the cost of safety is inevitably paid by the user experience.

Notes and Sources

  1. Röttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D. (2024). "XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models". NAACL 2024. ↩
  2. Askell, A. et al. (2021). "A General Language Assistant as a Laboratory for Alignment". arXiv:2112.00861. ↩
  3. Ouyang, L. et al. (2022). "Training language models to follow instructions with human feedback". arXiv:2203.02155. ↩
  4. Arditi, A., Obeso, O., Syed, A., Paleka, D., Rimsky, N., Gurnee, W., and Nanda, N. (2024). "Refusal in Language Models Is Mediated by a Single Direction". NeurIPS 2024. ↩
  5. Lin, Y. et al. (2024). "Mitigating the Alignment Tax of RLHF". EMNLP 2024. ↩
  6. Cui, J. et al. (2024). "OR-Bench: An Over-Refusal Benchmark for Large Language Models". arXiv:2405.20947; ICML 2025. ↩