Elamur Research⌕
☰
home / papers / 2025 / Hit the Sweet Spot! Span-Level Ensemble for Large Language Models

Hit the Sweet Spot! Span-Level Ensemble for Large Language Models

Paper · 2025 · Yangyifan Xu, Jianghao Chen, Junhong Wu et al.

Summary

Combining several large language models can exploit their complementary strengths, but current ensembling works at two awkward granularities. Sample-level methods pick or merge finished answers, so they cannot repair errors during generation. Token-level methods decide at every step, but each decision rests on too little evidence and must cope with mismatched tokenizers. The authors propose SweetSpan, a training-free method that ensembles short spans of whole words. In each round, every model continues the shared prefix with a fixed-length span. All models then score every candidate by perplexity. Extreme scores are discarded when they diverge strongly, and the span with the lowest averaged perplexity is appended. The method is tested with seven roughly 7B open chat models on NQ, GSM8K, MBPP and Flores-101 translation. It is evaluated in a standard setting (strongest models) and a new harder setting that mixes models of very uneven quality. SweetSpan beats sample- and token-level baselines in both settings, while earlier methods mostly break down in the harder one. This offers a vocabulary-independent way to combine heterogeneous LLMs without extra training.

Key points

  • Spans never cross word boundaries and have a predefined length (e.g., 4 words). This avoids tokenizer mismatch and balances having more evidence per decision against having more chances to correct errors.
  • Adaptive filtering removes outliers. If the highest perplexity for a span exceeds the lowest by a factor λ, both extremes are dropped before averaging. This stops weak models from boosting their own wrong spans or penalising correct ones.
  • Candidate models are LLaMA2-7B-Chat, ChatGLM2-6B, Baichuan2-7B-Chat, InternLM-7B-Chat, TigerBot-7B-Chat-V3, Vicuna-7B-V1.5 and ChineseAlpaca2-7B. Baselines include LLM-Blender and token-level ensembles.
  • Metrics are Exact Match (NQ), Accuracy (GSM8K), Pass@1 (MBPP) and BLEU (Flores-101 De↔En, Ro↔En).
  • The main cost is N inference passes plus perplexity computation, though both can run in parallel. The method is reported to be faster than token-level ensembling. Code is available at github.com/xydaytoy/SweetSpan.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🔬Institute of AutomationBeijing · China70 works · 14,251 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yangyifan Xu, Jianghao Chen, Junhong Wu and Jiajun Zhang. 2025. Hit the Sweet Spot! Span-Level Ensemble for Large Language Models. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{xu2025hit,
  title = {{Hit the Sweet Spot! Span-Level Ensemble for Large Language Models}},
  author = {Yangyifan Xu, Jianghao Chen, Junhong Wu and Jiajun Zhang},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.555/}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. Llm-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023). find
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  3. Baichuan. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
  4. Lingjiao Chen, Matei Zaharia, and James Zou. 2023a. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.
  5. Ye Chen, Wei Cai, Liangmin Wu, Xiaowei Li, Zhanxuan Xin, and Cong Fu. 2023b. Tigerbot: An open multilingual multitask llm. arXiv preprint arXiv:2312.08688.
  6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  7. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistic…
  8. Tianlin Li, Qian Liu, Tianyu Pang, Chao Du, Qing Guo, Yang Liu, and Min Lin. 2024. Purifying large language models by ensembling a small language model. arXiv preprint arXiv:2402.14845. — [arXiv:2402.14845](https://arxiv.org/abs/2402.14845) find
  9. Yuchen Liu, Long Zhou, Yining Wang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2018. A comparable study on model averaging, ensembling and reranking in nmt. In Natural Language Processing and Chinese Computing: 7th CCF International Conference, NLPCC 2018, Hohhot, China, August 26– 30, 2018, Proce… find
  10. Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. 2024. Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models. arXiv preprint arXiv:2407.06089. — [arXiv:2407.06089](https://arxiv.org/abs/2407.06089) find
  11. Long Zhou, Wenpeng Hu, Jiajun Zhang, and Chengqing Zong. 2017. Neural system combination for machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 378–384. find
  12. Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2023. Routing to the expert: Efficient reward-guided ensemble of large language models. arXiv preprint arXiv:2311.08692. find
  13. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics.
  14. Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. 2023. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789.
  15. InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM. ↗
  16. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  17. Omri Uzan, Craig W Schmidt, Chris Tanner, and Yuval Pinter. 2024. Greed is all you need: An evaluation of tokenizer inference methods. arXiv preprint arXiv:2403.01289. find
  18. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491. find
  19. Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. 2024. Bridging the gap between different vocabularies for llm ensemble. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7133–7… find
  20. Yao-Ching Yu, Chun-Chih Kuo, Ziqi Ye, Yu-Cheng Chang, and Yueh-Se Li. 2024. Breaking the ceiling of the llm community by treating token generation as a classification for ensembling. arXiv preprint arXiv:2406.12585. find
  21. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2025-hit-sweet-spot-span-level/