Elamur Research⌕
☰
home / papers / 2025 / Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly

Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly

Paper · 2025 · Peyman Hosseini, Ignacio Castro, Iacopo Ghinassi et al.

Summary

Modern LLMs advertise context windows that can hold hundreds of pages. This paper asks whether that capacity translates into reliable analysis of long documents on routine NLP tasks. The authors test Claude 3 Haiku, Gemini-1.0-Pro, GPT-3.5 Turbo, Llama 3 8b Instruct and Mistral 7b Instruct on three datasets. Sentiment analysis uses GameSpot reviews, and news categorization uses BBC News and 20 Newsgroups. Only documents above a minimum sentence count are kept.

They compare seven prompting scenarios:

  • feeding the full text, with or without an appended summary;
  • truncating to the first or last N sentences;
  • random sentence sampling;
  • a TextRank extractive summary;
  • a "diverse" summary that re-ranks TextRank candidates by TF-IDF cosine dissimilarity.

Across models and datasets, giving the whole document consistently performed worst. Short condensed inputs of about 7 sentences matched or beat it, even when the sentences were chosen at random. Condensing the input also cut cost and latency sharply. The work shows that a large context window does not guarantee good long-input understanding, and that cheap input reduction is a practical fix.

Key points

  • Datasets:
    • GameSpot: over 12,000 reviews, with 1–100 scores rounded into 10 classes.
    • BBC News Archive: 2,225 articles in 5 classes.
    • 20 Newsgroups: about 20,000 documents in 20 classes.
    • Average input lengths are roughly 2120, 1150 and 3450 tokens, respectively.
  • N choice: N = 7 sentences was used for all summary and truncation settings. An ablation showed gains flatten after about 7–8 sentences, and changes become marginal beyond 10.
  • GameSpot results: Diverse Summary reached 54.0% accuracy versus 36.0% for full context, a relative gain of about 50%.
  • News results:
    • 20 Newsgroups: Diverse Summary scored 39.5% accuracy versus 35.2% for full context.
    • BBC News: keeping the First Sentences worked best, at 64.5% versus 54.0%.
  • Efficiency:
    • Input shrank from about 3450 to 240 tokens on 20 Newsgroups, which cut API cost by up to 93%.
    • Average latency fell by about half (e.g., 1.58 s to 0.79 s).
    • All results are averaged over 5 runs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Queen Mary University of LondonLondon · United Kingdom54 works · 3,542 citations🔬Jožef Stefan InstituteLjubljana · Slovenia17 works · 2,165 citations🎓Jožef Stefan International Postgraduate SchoolLjubljana · Slovenia11 works · 1,776 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Peyman Hosseini, Ignacio Castro, Iacopo Ghinassi and Matthew Purver. 2025. Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hosseini2025efficient,
  title = {{Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly}},
  author = {Peyman Hosseini, Ignacio Castro, Iacopo Ghinassi and Matthew Purver},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.128/}
}

References

23 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Claude. https://www.anthropic.com. Version claude-3-haiku-20240307. Accessed: 2024-04-05 to 2024-06-11. ↗
  2. Bartomiej Balcerzak, Wojciech Jaworski, and Adam Wierzbicki. 2014. Application of textrank algorithm for credibility assessment. In 2014 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), volume 1, pages 451–454. IEEE. find
  3. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
  4. Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. 2024. Unlimiformer: Long-range transformers with unlimited length input. Advances in Neural Information Processing Systems, 36. find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Societal Impact and Ethical Considerations In terms of societal impact, we believe our findings can help enable the more effective and efficient application of LLMs to tasks involving longer doc- Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel Weld. 2020. TLDR: Extreme summarization of sci-uments,… find
  7. Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2022. A survey on dialogue summarization: Recent advances and new frontiers. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 5453–5460. International Joint Conferences on Artificial Intelligenc… find
  8. GameSpot. 2024. Gamespot reviews. Accessed: 2024-02-05 to 2024-06-11. find
  9. Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. In-context autoencoder for context compression in a large language model. In 12th International Conference on Learning Representations, ICLR. OpenReview.net. find
  10. Iacopo Ghinassi, Lin Wang, Chris Newell, and Matthew Purver. 2023. Comparing neural sentence encoders for topic segmentation across domains: not your typical text similarity task. PeerJ Computer Science, 9:e1593. find
  11. Derek Greene and Pádraig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proc. 23rd International Conference on Machine learning (ICML’06), pages 377–384. ACM Press. find
  12. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  13. Daniel Machlab and Rick Battle. 2024. Llm in-context recall is prompt dependent. arXiv preprint arXiv:2404.08865. — [arXiv:2404.08865](https://arxiv.org/abs/2404.08865) find
  14. Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404–411. Association for Computational Linguistics. find
  15. OpenAI. 2022. Gpt-3.5-turbo. https://openai.com/ blog/chatgpt. Accessed: 2024-03-10 to 2024-06-11. ↗
  16. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  17. Hei-Chia Wang, Wei-Ching Hsiao, and Sheng-Han Chang. 2020. Automatic paper writing based on a rnn and the textrank algorithm. Applied Soft Computing, 97:106767. find
  18. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  19. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023b. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376. Association for Computational L…
  20. Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning, pages 331–339. find
  21. Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. find
  22. Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060.
  23. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hosseini-2025-efficient-solutions-intriguing-failure-llms/