Elamur Research⌕
☰
home / papers / 2026 / Diagnosing LLMs via Information Spectrum Analysis: Tail Behavior and the Effects of Side Information

Diagnosing LLMs via Information Spectrum Analysis: Tail Behavior and the Effects of Side Information

Paper · 2026 · Yuuki Tachioka

Summary

Average likelihood scores such as perplexity can hide how differently an LLM behaves across samples, and failures tend to cluster among rare, highly surprising outputs. The paper adapts Han and Verdú's information spectrum (IS) theory, which describes general sources without needing stationarity, ergodicity or the AEP, into a practical diagnostic. For each generated sequence it computes a length-normalised mean surprisal (coding rate). Over a finite set of samples, these values form an empirical spectrum, summarised by upper and lower quantiles and the gap between them. A second tool, the information gain (IG) spectrum, rescores one fixed generated answer under teacher forcing, with and without side information, to give a per-sequence gain. The side information tested is answer options, RAG passages and knowledge graphs. Experiments on several Japanese LLMs in QA settings yield three findings. Correct and incorrect answers differ mainly among the most surprising sequences, not in the average. Side information helps sequences very unevenly. After instruction tuning, spread and tail statistics predict correctness better than the mean. The work argues for distribution-aware evaluation alongside averages.

Key points

  • The spectrum is summarised by empirical quantiles: Ĥθ sup is the (1−θ)-quantile and Ĥθ inf is the θ-quantile. Their difference is the width Ĥθ w, and a smaller θ emphasises the high-surprisal tail.
  • The IG spectrum is ΔZ = Z(t′; y′) − Z(t′; y). A negative ΔZ means the added context raises the likelihood of the same sequence. The author presents it as a controlled likelihood diagnostic, not a causal measure of how the generation distribution shifts.
  • Side information (multiple-choice options, RAG, KGs) reshapes the spectrum heterogeneously. In some cases it shrinks high-surprisal tails, and in others it enlarges them.
  • Correctness was judged automatically, by checking whether the normalised reference answer string appears in the output. The same rule was used for every model and condition, and the author acknowledges possible misclassifications.
  • Stated future directions include:
    • shape statistics and distributional distances, such as Wasserstein distance;
    • links to IS theory with side information (Kuzuoka and Watanabe, 2015);
    • more tasks and languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Denso IT Laboratory Inc.Tokyo · Japan2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksPsycholinguistics & language acquisition · 108 works

Cite

Cite as Yuuki Tachioka. 2026. Diagnosing LLMs via Information Spectrum Analysis: Tail Behavior and the Effects of Side Information. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.594

@inproceedings{tachioka2026diagnosing,
  title = {{Diagnosing LLMs via Information Spectrum Analysis: Tail Behavior and the Effects of Side Information}},
  author = {Yuuki Tachioka},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.594},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.594}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rudolf Ahlswede, Lars Bäumer, Ning Cai, Harout Ay-dinian, Vladimir Blinovsky, Christian Deppe, and Haik Mashurian, editors. 2006. General Theory of Information Transfer and Combinatorics, 1 edition. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg. find
  2. Tomoki Ariyama, Jun Suzuki, Masatoshi Suzuki, Ryota Tanaka, Reina Akama, and Kyosuke Nishida. 2024. Achievements and challenges in Japanese question answering: Insights from quiz competition results. Natural Language Processing, 31(1):47–78. find
  3. Stefania Degaetano-Ortlieb and Elke Teich. 2017. Modeling intra-textual variation with entropy and surprisal: topical vs. stylistic patterns. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 68–77, Vanco… find
  4. Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. 2025. What is wrong with perplexity for long-context language modeling? Preprint, arXiv:2410.23771. — [arXiv:2410.23771](https://arxiv.org/abs/2410.23771) find
  5. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  6. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual LLM adaptation: Enhancing Japanese language capabilities. In First Conference on Language Modeling.
  7. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language…
  8. Te Sun Han. 2003. Information-Spectrum Methods in Information Theory, 1 edition. Stochastic Modelling and Applied Probability. Springer Berlin, Heidelberg. Original Japanese edition published by Bai-fukan, Tokyo, 1998. find
  9. T.S. Han and S. Verdu. 1993. Approximation theory of output statistics. IEEE Transactions on Information Theory, 39(3):752–772. find
  10. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
  11. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2).
  12. Ai Ishii, Naoya Inoue, Hisami Suzuki, and Satoshi Sekine. 2024. JEMHopQA: Dataset for Japanese explainable multi-hop question answering. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9515–9525, Tori… find
  13. Jiaming Ji, Kaile Wang, Tianyi Alex Qiu, Boyuan Chen, Jiayi Zhou, Changye Li, Hantao Lou, Josef Dai, Yunhuai Liu, and Yaodong Yang. 2025. Language models resist alignment: Evidence from data compression. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volu… find
  14. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
  15. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022.… find
  16. Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. JGLUE: Japanese general language understanding evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2957–2966, Marseille, France. European Language Resources Association.
  17. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego, California. Association for Computational Linguistics. find
  18. Hongfu Liu and Ye Wang. 2023. Towards informative few-shot prompt with maximum information gain for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15825–15838, Singapore. Association for Computational Linguistics. find
  19. Nguyen Nhat Minh, Andrew Baker, Clement Neo, Allen G Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. 2025. Turning up the heat: Min-p sampling for creative and coherent LLM outputs. In The Thirteenth International Conference on Learning Representations. find
  20. CE Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27:379–423, 623–656. find
  21. ByungHoon So, Kyuhong Byun, Kyungwon Kang, and Seongjin Cho. 2022. JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension. Preprint, arXiv:2202.01764. — [arXiv:2202.01764](https://arxiv.org/abs/2202.01764) find
  22. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  23. S. Verdu and Te Sun Han. 1994. A general formula for channel capacity. IEEE Transactions on Information Theory, 40(4):1147–1157. find
  24. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  25. Shigeaki Kuzuoka and Shun Watanabe. 2015. An information-spectrum approach to weak variable-length source coding with side-information. IEEE Transactions on Information Theory, 61(6):3559– 3573. find
  26. Sohee Yang, Jonghyeon Kim, Joel Jang, Seonghyeon Ye, Hyunji Lee, and Minjoon Seo. 2024. Improving probability-based prompt selection through unified evaluation and analysis. Transactions of the Association for Computational Linguistics, 12:664–680. find
  27. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th Internation…
  28. A Coding-Theoretic Interpretation of Information-Spectrum Quantities Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In sup / inf

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tachioka-2026-diagnosing-llms-information-spectrum-analysis/