Elamur Research⌕
☰
home / papers / 2024 / Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models

Summary

LLMs advertise ever-larger context windows, but it is unclear whether a model that solves a task on a short prompt can still solve it when the same task sits inside a longer one. Earlier long-context benchmarks changed the task and the length together, so the two effects could not be separated. Levy, Jacoby and Goldberg isolate length with FLenQA, a True/False reasoning dataset. Each item needs two key facts to be combined. The authors keep those facts fixed and surround them with irrelevant padding of different sizes, sources and positions. On average across the tested models, accuracy falls from 0.92 to 0.68 at only about 3000 tokens, far below the models' maximum context sizes. The decline appears in every padding variant, though its strength varies. Chain-of-Thought prompting raises accuracy overall but generally does not close the length-related gap. Next-word prediction quality also fails to predict long-input reasoning. The work argues that a long-context model must be evaluated across all the lengths it claims to support.

Key points

  • FLenQA has three tasks, each with 100 base instances and balanced True/False labels:
    • Monotone Relations (MonoRel), a new task;
    • People In Rooms (PIR), a new task inspired by bAbI;
    • a simplified Ruletaker.
  • Key sentences were expanded into coherent paragraphs with GPT-4 and then checked by hand. Person names come from the Faker library. Code to regenerate the data from scratch is released to limit contamination.
  • The evaluated models are GPT3.5 (gpt-3.5-turbo-1106), GPT4 (gpt-4-1106-preview), Gemini Pro, Mistral Medium and Mixtral 8x7B.
  • Chain-of-Thought gives a roughly constant gain at every length. GPT4 is the only model whose advantage from CoT widens as inputs grow.
  • Several failure modes grow with input length:
    • a bias toward answering "False";
    • responses that give no answer at all;
    • stating the answer before the reasoning steps (odds-ratio 3.643 for incorrect answers);
    • weaker coverage of the key facts in CoT output (odds-ratio 3.138).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

102 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 99 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.818

@inproceedings{levy2024same,
  title = {{Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models}},
  author = {Levy, Mosh and Jacoby, Alon and Goldberg, Yoav},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.818},
  url = {https://doi.org/10.18653/v1/2024.acl-long.818}
}

References

33 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chen An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023a. Leval: Instituting standardized evaluation for long context language models. ArXiv, abs/2307.11088. find our card
  2. Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023b. L-eval: Instituting standardized evaluation for long context language models. find our card
  3. Rohan Anil, Sebastian Borgeaud, Yonghui Wu, and Gemini Team Google. 2023. Gemini: A family of highly capable multimodal models. ArXiv, abs/2312.11805. find our card
  4. Yushi Bai, Xin Lv, Jiajie Zhang, Hong Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. ArXiv, abs/2308.14508. find our card
  5. Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). find
  6. Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2021. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3882–3890. find our card
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. find our card
  8. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051. find
  9. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrope: Extending llm context window beyond 2 million tokens. find our card
  10. Daniele Faraglia and Other Contributors. 2012. Faker. find
  11. Alexios Gidiotis and Grigorios Tsoumakas. 2020. A divide-and-conquer approach to the summarization of long documents. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:3029–3040. find
  12. Ari Holtzman, Peter West, and Luke Zettlemoyer. 2023. Generative models as a complex systems science: How can we make sense of large language model behavior? arXiv preprint arXiv:2308.00189. find
  13. Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403. find our card
  14. Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5075–5084, Sing… find
  15. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  16. Mingyu Jin, Qinkai Yu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, Mengnan Du, et al. 2024. The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.04925. find our card
  17. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213. find our card
  18. Mosh Levy, Shauli Ravfogel, and Yoav Goldberg. 2023. Guiding llm to fool itself: Automatically manipulating machine reading comprehension shortcut triggers. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8495–8505. find
  19. Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023. Loogle: Can long-context language models understand long contexts? ArXiv, abs/2311.04939. find our card
  20. Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. 2023a. Same pre-training loss, better downstream: Implicit bias matters for language models. In International Conference on Machine Learning, pages 22188–22214. PMLR. find
  21. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023b. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. find our card
  22. Yang Liu, Chenguang Zhu, and Michael Zeng. 2022. End-to-end segmentation-based news summarization. In Findings of the Association for Computational Linguistics: ACL 2022, pages 544–554. find
  23. Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020. Generation-augmented retrieval for open-domain question answering. In Annual Meeting of the Association for Computational Linguistics. find our card
  24. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In Annual Meeting of the Association for Computational Linguistics. find our card
  25. OpenAI. 2023. Gpt-4 technical report. find our card
  26. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787. find our card
  27. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305.14196. find our card
  28. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022. Scrolls: Standardized comparison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1200… find our card
  29. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR. find our card
  30. Koustuv Sinha, Shagun Sodhani, William L. Hamilton, and Joelle Pineau. 2018. Compositional language understanding with text-based relational reasoning. ArXiv, abs/1811.02959. — [arXiv:1811.02959](https://arxiv.org/abs/1811.02959) find
  31. Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. 2022. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551. find
  32. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837. find our card
  33. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Representations, ICLR 2016. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/levy-2024-same-task-more-tokens-impact/