Elamur Research⌕
☰
home / papers / 2026 / What Tokens Truly Matter? The Logit Conflation Problem in LLM Sampling

What Tokens Truly Matter? The Logit Conflation Problem in LLM Sampling

Paper · 2026 · Pinlong Zhao, Huijun Tang, Pengfei Jiao et al.

Summary

Standard LLM samplers such as top-k, top-p, min-p and top-nσ pick candidate tokens by logit size, assuming a high logit means a good choice. The authors argue this is flawed for instruction following, which they call the Logit Conflation Problem. A single logit mixes general fluency, associations learned in the parameters, and actual relevance to the prompt, and only the last of these predicts whether the instruction is satisfied. They define prompt-relevance as the change in a token's logit when the prompt is present versus masked. Computing that directly would need an extra forward pass, so they approximate it cheaply. Final-layer attention pools the prompt's hidden states, and each candidate's embedding is scored against that pooled vector. A gradient-derived attribution score then removes high-logit tokens that are irrelevant to the prompt. The resulting decoder, SEAL-Sampling, needs no training and beats top-nσ on instruction-following benchmarks with LLaMA-3 and Qwen2. This suggests decoding could target what the prompt asks rather than raw token likelihood.

Key points

  • SEAL-Sampling (Signal Extraction for Active ReLevance) has three stages:
    • statistical pre-filtering of candidates;
    • relevance scoring against the attention-weighted prompt representation;
    • filtering by an attribution score of the form p_i(1−p_i)·max(0, r_i).
  • On LLaMA-3, it improves over top-nσ by 1.8% on AlpacaEval 2.0 and by 2.2% on IFEval. Gains are also reported for Qwen2-7B and grow as prompts become more constrained.
  • The cost is O(kd) per step, adding under 12 ms per token, and it uses a single forward pass.
  • The proxy correlates about 0.60 with true prompt-relevance, versus 0.35 for raw logits. On 500 IFEval examples, fluency varies far less than relevance among filtered candidates (CV 0.12 vs 0.87).
  • Limitations:
    • it requires access to model internals, so it cannot run through API-only models;
    • it was evaluated only on English;
    • gains are smaller on math and code tasks;
    • performance drops slightly on very long prompts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hangzhou Dianzi UniversityHangzhou · China15 works · 1,061 citations🎓Durham UniversityDurham · United Kingdom9 works · 561 citations🎓Tianjin Normal UniversityTianjin · China3 works · 11 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Pinlong Zhao, Huijun Tang, Pengfei Jiao and Mengyang Li. 2026. What Tokens Truly Matter? The Logit Conflation Problem in LLM Sampling. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1841

@inproceedings{zhao2026what,
  title = {{What Tokens Truly Matter? The Logit Conflation Problem in LLM Sampling}},
  author = {Pinlong Zhao, Huijun Tang, Pengfei Jiao and Mengyang Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1841},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1841}
}

References

25 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. In ACL.
  2. David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. 1985. A learning algorithm for Boltzmann machines. Cognitive Science. find
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  4. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does BERT Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In NAACL.
  5. Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. GeDi: Generative discriminator guided sequence generation. In EMNLP Findings.
  6. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In SOSP.
  7. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  8. Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In EMNLP Findings. find
  9. Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. 2021. DExperts: Decoding-time controlled text generation with experts and anti-experts. In ACL.
  10. Minh Nguyen, Andrew Baker, Clement Neo, Allen Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. 2024. Turning up the heat: Min-p sampling for creative and coherent llm outputs. arXiv preprint arXiv:2407.01082. — [arXiv:2407.01082](https://arxiv.org/abs/2407.01082) find
  11. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In EMNLP.
  12. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, and 1 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  13. Kevin Yang and Dan Klein. 2021. FUDGE: Controlled text generation with future discriminators. In NAACL.
  14. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS.
  15. Jeffrey Zhou and 1 others. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
  16. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. In SIGdial.
  17. Sean O’Brien and Mike Lewis. 2023. Contrastive decoding improves reasoning in large language models. arXiv preprint arXiv:2309.09117.
  18. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. In NeurIPS.
  19. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR Workshop.
  20. Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In NeurIPS, volume 35, pages 21548–21561. find
  21. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In ICML. find
  22. Chenxia Tang, Jianchun Liu, Hongli Xu, and Liusheng Huang. 2025. Top-nσ: Eliminating noise in logit space for robust token sampling of llm. In ACL). find
  23. Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. arXiv preprint arXiv:1906.04284.
  24. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022a. Finetuned language models are zero-shot learners. In ICLR.
  25. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022b. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-what-tokens-truly-matter-logit/