Elamur Research⌕
☰
home / papers / 2026 / What Matters to an LLM? Behavioral and Computational Evidences from Summarization

What Matters to an LLM? Behavioral and Computational Evidences from Summarization

Paper · 2026 · Yongxin Zhou, Changshun Wu, Philippe Mulhem et al.

Summary

LLMs now lead in summarization, but it is unclear what internal sense of importance decides which content they keep. The authors study this from two sides. Behaviorally, they summarize each document ten times under length budgets from 10 to 100. A token's importance is the fraction of summaries that contain it. They apply this to 7 LLMs on CNN/DailyMail, SAMSum and DECODA-French. Computationally, they compare these distributions with aggregated attention weights. They also train low-capacity linear probes on hidden states, with two controls: probes on randomized-weight models (a "dead salmon" check) and a TextRank baseline.

LLMs largely agree with each other on what matters and depart clearly from pre-LLM methods. They group by model family more than by size. Certain attention heads, and especially middle-to-late layers, predict importance well. Predictability is highest for dialogues and lowest for news. The work is a first step toward interpreting, and later controlling, how LLMs select information.

Key points

  • Released resource: 274,330 summaries from 7 LLMs, preprocessed into token-level importance distributions. Code and data are on GitHub (yongxin2020/llm-importance-summarization).
  • Importance is defined as how often a unit persists across 10 length-controlled summaries. The idea is related to Pyramid-style frequency counting.
  • In article-level probing, probes on trained weights beat randomized-weight probes by about 0.1–0.2 NDCG@10 and consistently surpass TextRank. Performance peaks in middle-to-late layers.
  • Importance encoding shows no consistent gain with model scale. Llama models outperform Qwen, and SAMSum is far more predictable than CNN/DailyMail.
  • A case study on Qwen2.5-3B-Instruct shows named entities scoring high in news and participant names scoring high in dialogues.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSummarization · 232 works

Cite

Cite as Yongxin Zhou, Changshun Wu, Philippe Mulhem, Didier Schwab and Maxime Peyrard. 2026. What Matters to an LLM? Behavioral and Computational Evidences from Summarization. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.302

@inproceedings{zhou2026what,
  title = {{What Matters to an LLM? Behavioral and Computational Evidences from Summarization}},
  author = {Yongxin Zhou, Changshun Wu, Philippe Mulhem, Didier Schwab and Maxime Peyrard},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.302},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.302}
}

References

55 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. 2024. Llama models card. find
  2. Jie Bao, Prithwish Basu, Mike Dean, Craig Partridge, Ananthram Swami, Will Leland, and James A. Hendler. 2011. Towards a theory of semantic communication. 2011 IEEE Network Science Workshop, pages 110–117. find
  3. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations. find
  4. Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. 2020. Thread: Circuits. Distill. Https://distill.pub/2020/circuits. find
  5. Rudolf Carnap and Yehoshua Bar-Hillel. 1952. An outline of a theory of semantic information. find
  6. Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. 2019. Activation atlas. Distill. find
  7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2… find
  8. Zhichen Dong, Zhanhui Zhou, Zhixuan Liu, Chao Yang, and Chaochao Lu. 2025. Emergent response planning in LLMs. In Forty-second International Conference on Machine Learning. find
  9. Benoit Favre, Evgeny Stepanov, Jérémy Trione, Frédéric Béchet, and Giuseppe Riccardi. 2015. Call centre conversation summarization: A pilot task at multiling 2015. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 232–236, Prague, Czech Republic… find
  10. Mohamed Imed Eddine Ghebriout, Gaël Guibon, Ivan Lerner, and Emmanuel Vincent. 2025. QUARTZ: QA-based unsupervised abstractive refinement for task-oriented dialogue summarization. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 14689–14706, Suzhou, China. Association… find
  11. Daniel Gillick, Benoit Favre, and Dilek Hakkani-Tür. 2008. The icsi summarization system at tac 2008. In Tac. find
  12. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.
  13. Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1449–1462, Online. Association for Computatio…
  14. Tianjie Ju, Zhenyu Shao, Bowen Wang, Yujia Chen, Zhuosheng Zhang, Hao Fei, Mong-Li Lee, Wynne Hsu, Sufeng Duan, and Gongshen Liu. 2025. Probing then editing response personality of large language models. In Second Conference on Language Modeling. find
  15. Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818–1828, Brussels, Belgium. Association for Computational Linguistics.
  16. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057–7075, Online. Association for Computational…
  17. Abhinav Kumar, Chenhao Tan, and Amit Sharma. 2022. Probing classifiers are unreliable for concept removal and detection. Advances in Neural Information Processing Systems, 35:17994–18008. find
  18. Chen Li, Xian Qian, and Yang Liu. 2013. Using supervised bigram-based ILP for extractive summarization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1004–1013, Sofia, Bulgaria. Association for Computational Linguistics. find
  19. Raymond Li, Wen Xiao, Linzi Xing, Lanjun Wang, Gabriel Murray, and Giuseppe Carenini. 2022. Human guided exploitation of interpretable attention patterns in summarization and topic segmentation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10189–10… find
  20. Wei Li, Lei He, and Hai Zhuge. 2016. Abstractive news summarization based on event semantic link network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 236–246, Osaka, Japan. find
  21. The COLING 2016 Organizing Committee. find
  22. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  23. Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of research and development, 2(2):159–165. find
  24. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.
  25. Ryan McDonald. 2007. A study of global inference algorithms in multi-document summarization. In Proceedings of the 29th European Conference on IR Research, ECIR’07, page 557–564, Berlin, Heidelberg. Springer-Verlag. find
  26. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36. ArXiv:2202.05262. find
  27. Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404–411, Barcelona, Spain. Association for Computational Linguistics. find
  28. Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kiciman, Hamid Palangi, Barun Patra, and Robert West. 2024. A glitch in the matrix? locating and detecting language model grounding with fakepedia. In Proceedings of the 62nd Annual Meeting of the Association for… find
  29. Maxime Méloux, Giada Dirupo, François Portet, and Maxime Peyrard. 2025. The dead salmons of ai interpretability. Preprint, arXiv:2512.18792. — [arXiv:2512.18792](https://arxiv.org/abs/2512.18792) find
  30. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As…
  31. Ani Nenkova and Kathleen McKeown. 2012. A Survey of Text Summarization Techniques, pages 43–76. Springer US, Boston, MA. find
  32. Ani Nenkova, Rebecca Passonneau, and Kathleen McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Trans. Speech Lang. Process., 4(2):4–es. find
  33. Ani Nenkova and Lucy Vanderwende. 2005. The impact of frequency on summarization. Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005, 101. find
  34. Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. 2025. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. In The Thirteenth International Conference on Learning Representations. find
  35. Maxime Peyrard. 2019. A simple theoretical model of importance for summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1059–1073, Florence, Italy. Association for Computational Linguistics. find
  36. Maxime Peyrard and Judith Eckle-Kohler. 2016. Optimizing an approximation of ROUGE - a problem-reduction approach to extractive multi-document summarization. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1825–1836, Berlin, G… find
  37. Sanjana Ramprasad, Elisa Ferracane, and Zachary Lipton. 2024. Analyzing LLM behavior in dialogue summarization: Unveiling circumstantial hallucination trends. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12549–12561, Bangko… find
  38. Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3363–3377, Online. Association…
  39. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Compu… find
  40. Julius Steen and Katja Markert. 2024. Bias in news summarization: Measures, pitfalls and corpora. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5962–5983, Bangkok, Thailand. Association for Computational Linguistics. find
  41. Xu Sun, Lionel Delphin-Poulat, Christèle Tarnec, and Anastasia Shimorina. 2025. PoSum-bench: Benchmarking position bias in LLM-based conversational summarization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7996–8020, Suzhou, China. Association fo… find
  42. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  43. Damien Teney, Maxime Peyrard, and Ehsan Abbasnejad. 2022. Predicting is not understanding: Recognizing and addressing underspecification in machine learning. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIII, page 458–476, Berlin… find
  44. Jan Trienes, Jörg Schlötterer, Junyi Jessy Li, and Christin Seifert. 2025. Behavioral analysis of information salience in large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 23428–23454, Vienna, Austria. Association for Computational Linguistics. find
  45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  46. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I…
  47. David Wan, Jesse Vig, Mohit Bansal, and Shafiq Joty. 2025. On positional bias of faithfulness for long-form summarization. In Proceedings of the 2025 find
  48. Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8791–8810, Albuquerque, New Mexico. Association for Computational Linguistics. find
  49. Linzi Xing, Wen Xiao, and Giuseppe Carenini. 2021. Demoting the lead bias in news summarization via alternating adversarial learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing… find
  50. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57.
  51. Yixin Zhong. 2017. A theory of semantic information. Proceedings, 1(3). find
  52. Yongxin Zhou, François Portet, and Fabien Ringeval. 2022. Effectiveness of French language models on abstractive dialogue summarization task. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3571–3581, Marseille, France. European Language Resources Association. find
  53. Yongxin Zhou, Fabien Ringeval, and François Portet. 2024. PSentScore: Evaluating sentiment polarity in dialogue summarization. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13290–13302, Torino, Ital… find
  54. Yongxin Zhou, Fabien Ringeval, and François Portet. 2025. Can GPT models follow human summarization guidelines? a study for targeted communication goals. In Proceedings of the 18th International Natural Language Generation Conference, pages 249– 273, Hanoi, Vietnam. Association for Computational Lin… find
  55. Markus Zopf, Eneldo Loza Mencía, and Johannes Fürnkranz. 2016. Beyond centrality and structural features: Learning information importance for text summarization. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 84–94, Berlin, Germany. Association for Com… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-what-matters-llm-behavioral-computational/