Elamur Research⌕
☰
home / papers / 2026 / SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs

SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs

Paper · 2026 · Jie Sun, Yu Liu, Lu Han et al.

Summary

Modern LLMs advertise very long context windows, yet they handle long sequences of numbers poorly. Tasks such as counting, filtering, or spotting trends need exact access to every value. The authors trace this failure to attention dispersion: as inputs grow, Softmax spreads attention over many similar numeric tokens, so focus on local values weakens. Tool use and Program-of-Thought do not remove the problem, because the model must still copy the sequence correctly. The authors propose SepSeq, a training-free, plug-and-play input-formatting method. It inserts a separator token (e.g., a newline) after every k elements, splitting the sequence into short segments. Attention analysis suggests these separators act as attention sinks, which concentrates attention within segments while the full sequence remains reachable. Across 9 LLMs, 6 synthetic tasks and 4 real numerical domains, SepSeq improves accuracy by 35.6% on average (relative) and also shortens outputs. This makes it a cheap, practical fix for numeric-heavy workloads such as finance and time-series question answering.

Key points

  • In a verbatim-copying test (Appendix A), success falls to 14% for sequences longer than 256 floating-point numbers. The authors argue that this limits tool-based workarounds.
  • SepSeq is complementary to Program-of-Thought (10 tasks, 200 samples each):
    • Standard prompting rises from 53.25% to 70.35%.
    • PoT rises from 78.90% to 84.45%, the best combination.
  • Reasoning ("Thinking") models gain more than Instruct models:
    • Thinking models go from 71.4% to 90.2%.
    • Instruct models go from 57.3% to 72.1%.
  • Average response length drops from 3,375 to 2,823 tokens (−16.4%). Reductions are largest on Indexing (~42.9%) and Stock (~29.6%), attributed to fewer repetition loops.
  • Limitation: on the Qwen3 family, small models such as 0.6B perform worse with SepSeq. Gains become robust only well above roughly 4B parameters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations🎓Nanjing UniversityNanjing · China74 works · 7,814 citations🎓East China Normal UniversityShanghai · China50 works · 2,174 citations🎓Shanghai Innovation InstituteShanghai · China24 works · 262 citations🎓Ocean University of ChinaQingdao · China2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Jie Sun, Yu Liu, Lu Han, Qiwen Deng, Xiang Shu, Yang Xiao, Lintao Ma, Xingyu Lu, Jun Zhou, Pengfei Liu, Jiancan Wu and Xiang Wang. 2026. SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.272

@inproceedings{sun2026sepseq,
  title = {{SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs}},
  author = {Jie Sun, Yu Liu, Lu Han, Qiwen Deng, Xiang Shu, Yang Xiao, Lintao Ma, Xingyu Lu, Jun Zhou, Pengfei Liu, Jiancan Wu and Xiang Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.272},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.272}
}

References

50 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint, abs/2303.08774.
  2. Murtadha H. M. Ahmed, Wen Bo, and Yunfeng Liu. 2025. MateICL: Mitigating attention dispersion in large-scale in-context learning. arXiv preprint, abs/2505.01110. — [arXiv:2505.01110](https://arxiv.org/abs/2505.01110) find
  3. Anthropic. 2025. System card: Claude opus 4 & claude sonnet 4. find
  4. Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João G.M. Araújo, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković. 2024. Transformers need glasses! Information over-squashing in language tasks. In Advances in Neural Information Processing Systems 37, pages 98111– 98142… find
  5. Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. 2024. SepLLM: Accelerate large language models by compressing one segment into one separator. In Proceedings of the 41st International Conference on Machine Learning, Vienna, A… find
  6. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023a. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. find
  7. Trans. Mach. Learn. Res., 2023. find
  8. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023b. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023. find
  9. Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, Vienna, Austria. find
  10. Yihong Dong, Yuchen Liu, Xue Jiang, Bin Gu, Zhi Jin, and Ge Li. 2025. Rethinking repetition problems of LLMs in code generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 965–985, Vienna, Austria. find
  11. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and fate: Limits of transformers on composition…
  12. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformer: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23:1–39. find
  13. Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. ReTool: Reinforcement learning for strategic tool use in LLMs. arXiv preprint, abs/2504.11536. find
  14. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, pages 10764– 10799, Honolulu, HI. find
  15. Gemini-Team. 2025. Gemini 2.5 pro model card. https://storage.googleapis.com/model-cards/documents/gemini-2.5-pro.pdf. ↗
  16. GLM-4.5-Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, and 152 others. 2025. GLM-4.5: Agentic, Reasoning, and Coding (ARC)…
  17. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint, abs/2501.12948.
  18. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, New Orleans…
  19. Peyman Hosseini, Ignacio Castro, Iacopo Ghinassi, and Matthew Purver. 2025. Efficient solutions for an intriguing failure of LLMs: Long context window does not mean LLMs can analyze long sequences flawlessly. In Proceedings of the 31st International Conference on Computational Linguistics, pages 188…
  20. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: what’s the real context size of your long-context language models? In Proceedings of the 2024 Conference on Language Modeling, Philadelphia, PA.
  21. Xinyan Jiang, Hang Ye, Yongxin Zhu, Xiaoying Zheng, Zikang Chen, and Jun Gong. 2025. HICD: Hallucination-inducing via attention dispersion for contrastive decoding to mitigate hallucinations in large language models. In Findings of the Association for Computational Linguistics, pages 7764– 7786, Vie… find
  22. Kimi-Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, and 150 others. 2025. Kimi K2: Open agentic intelligence. arXiv prepr…
  23. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint, abs/1808.06226.
  24. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Y. Sorokin, and Mikhail Burtsev. 2024. BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack. In Advances in Neural Information Processing Systems 38, Vancouver, Canada. find
  25. Haoyang Li, Xuejia Chen, Zhanchao Xu, Darian Li, Nicole Hu, Fei Teng, Yiming Li, Luyu Qiu, Chen Jason Zhang, Qing Li, and Lei Chen. 2025. Exposing numeracy gaps: A benchmark to evaluate fundamental numerical abilities in large language models. In Findings of the 63th Annual Meeting f the Association… find
  26. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  27. Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. 2023. Scaling laws of rope-based extrapolation. arXiv preprint, abs/2310.05209.
  28. Meta-AI. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/. ↗
  29. Ken M. Nakanishi. 2025. Scalable-softmax is superior for attention. arXiv preprint, abs/2501.19399. — [arXiv:2501.19399](https://arxiv.org/abs/2501.19399) find
  30. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint, abs/2309.00071. find
  31. Tiago Pimentel and 1 others. 2025. Repetitions are not all alike: Distinct mechanisms sustain repetition in language models. arXiv preprint, abs/2504.01100. — [arXiv:2504.01100](https://arxiv.org/abs/2504.01100) find
  32. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. 2025. Toolrl: Reward is all tool learning needs. arXiv preprint, abs/2504.13958. find
  33. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2025. Tool learning with large language models: a survey. Frontiers of Computer Science, 19(8):198343.
  34. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint, abs/2302.04761.
  35. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems 36, New Orleans, LA.
  36. John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive Science, 12:257– 285. find
  37. John Sweller, Paul Ayres, and Slava Kalyuga. 2011. Cognitive Load Theory. Springer, New York. find
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint, abs/1706.03762.
  39. Petar Velickovic, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. 2024. Softmax is not enough (for sharp size generalisation). arXiv preprint, abs/2410.01104. — [arXiv:2410.01104](https://arxiv.org/abs/2410.01104) find
  40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.
  41. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019. Neural text generation with unlikelihood training. arXiv preprint, abs/1908.04319.
  42. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In Proceedings of the the 12th International Conference on Learning Representations, Vienna, Austria. find
  43. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint, abs/2505.09388.
  44. Junchi Yao, Shu Yang, Jianhua Xu, Lijie Hu, Mengdi Li, and Di Wang. 2025. Understanding the repeat curse in large language models from a feature perspective. arXiv preprint, abs/2504.14218. find
  45. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations, Kigali, Rwanda.
  46. Haizi Yu, Igor Mineyev, Lav R. Varshney, and James A. Evans. 2022. Learning from one and only one shot. arXiv preprint, abs/2201.08815. — [arXiv:2201.08815](https://arxiv.org/abs/2201.08815) find
  47. Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Manoj Awalgaonkar, Rithesh R. N., Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, and 2 others. 2025. xlam: A family…
  48. Yiming Zhang, Shi Li, and Yang Liu. 2022. On the robustness of chinese text processing pipelines. Findings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3548–3560. find
  49. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jianfeng Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, and 1 others. 2023. Mixture-of-experts meets instruction tuning: A winning combination for large language models. arXiv preprint, abs/2305.14705.
  50. Pál Zsámboki, Benjamin Levi, David Ansel Josef Smith, Mitansh Kagalwala, Arlington Kell, Samuel Liechty, and Cong Wang. 2025. Learning what’s missing: Attention dispersion and EMA stabilization in length generalization. arXiv preprint, abs/2510.08341. — [arXiv:2510.08341](https://arxiv.org/abs/2510.08341) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-sepseq-training-free-framework-long/