Elamur Research⌕
☰
home / papers / 2026 / Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms

Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms

Paper · 2026 · Xinlin Wang

Summary

Organizations under strict privacy rules (e.g., GDPR, PCI DSS) and tight hardware budgets often cannot use cloud LLMs. They must rely on open models under roughly 10B parameters, which are weak at multi-step numerical reasoning and domain knowledge. Instead of scaling or fine-tuning, Wang tests whether system architecture can close this gap. The study benchmarks 27 open-source SLMs from seven families in three setups:

  • plain zero-shot prompting
  • a ReAct-style single agent with tools
  • a supervisor that routes work to three specialist agents

Evaluation covers 20 public financial datasets across 8 task types. Besides answer quality, it tracks completion rate, latency and energy use. The tool-equipped single agent gives the best quality-to-cost ratio. The multi-agent variant adds coordination overhead, instability and energy cost without matching gains. Plain prompting still wins on classification and token-level tasks. The work matters because it offers practical deployment guidance for teams that cannot run large models, favouring architectural choice over parameter count.

Key points

  • Models: 27 open-source models from Qwen, Llama, Gemma, Phi, DeepSeek-R1 distills, Mistral and SOLAR, sized from 270M (gemma-3-270m) to SOLAR 10.7B. The largest model slightly exceeds the stated sub-10B scope.
  • Data: 20 financial datasets in 8 categories (sentiment, classification, NER, QA, stock movement, credit scoring, summarization, bankruptcy), with 50 sampled instances per dataset.
  • Paradigms:
    • The single agent can call a calculator, wiki search and web search.
    • The multi-agent system has a tool-less supervisor plus FinancialNLPAgent, FinancialKnowledgeAgent and FinancialQuantAgent, each with restricted tools.
    • Metrics include Completion Rate, Normalized Response Quality (NRQ) and average latency.
  • Results:
    • On generative tasks, the single agent improved some models by more than 150% (e.g., Qwen2.5-0.5B, Llama-2-7b).
    • Gemma and Phi models often did best with base prompting on NER, sentiment and stock prediction.
    • Agent setups for gemma-3-270m mostly failed to run.
    • Base model performance does not rise steadily with size; the paper reports a "U-turn" pattern.
  • Failure modes:
    • Single agents mostly broke on exceeded context length and missing delegation.
    • Multi-agent runs suffered delegation failures.
    • Malformed structured outputs and wrong tool choices were common in both.
    • Reasoning and tool effects are not separated in the analysis. The author advises falling back to the base model when an agent stalls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Xinlin Wang. 2026. Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.123

@inproceedings{wang2026rethinking,
  title = {{Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms}},
  author = {Xinlin Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.123},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.123}
}

References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wei-Lin Chiang and 1 others. 2023. Open llm leaderboard. In NeurIPS Datasets and Benchmarks Track. find
  2. Tri Dao and 1 others. 2022. Flashattention: Fast and memory-efficient exact attention. In Advances in Neural Information Processing Systems (NeurIPS). find
  3. Tim Dettmers and 1 others. 2023. Efficient fine-tuning of small language models. In International Conference on Machine Learning (ICML). find
  4. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning.
  5. Committee on Economic European Parliament, Monetary Affairs, and Rapporteur Arba Kokalari. 2025. Report on the impact of artificial intelligence on the financial sector (a10-0225/2025). Adopted 5.11.2025 by the Committee on Economic and Monetary Affairs. find
  6. Mohd Ariful Haque, Fahad Rahman, Kishor Datta Gupta, Khalil Shujaee, and Roy George. 2025. Tinyllm: Evaluation and optimization of small language models for agentic tasks on edge devices. Preprint, arXiv:2511.22138. find
  7. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  8. Percy Liang, Rishi Bommasani, Tony Lee, and 1 others. 2022. Holistic evaluation of language models. In Advances in Neural Information Processing Systems (NeurIPS). find
  9. Nghiem Thanh Pham, Tung Kieu, Duc Manh Nguyen, Son Ha Xuan, Nghia Duong-Trung, and Danh Le-Phuoc. 2025. SLM-bench: A comprehensive benchmark of small language models on environmental impacts. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 21369– 21392, Suzhou, China.… find
  10. Michael Rizvi-Martel, Satwik Bhattamishra, Neil Rathi, Guillaume Rabusseau, and Michael Hahn. 2025. Benefits and limitations of communication in multi-agent reasoning. Preprint, arXiv:2510.13903. — [arXiv:2510.13903](https://arxiv.org/abs/2510.13903) find
  11. Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. 2024. Small LLMs are weak tool learners: A multi-LLM agent. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16658–16680, Miami, Florida, USA. Asso…
  12. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366, 1. ↗
  13. Bo Wang, Weiyi He, Shenglai Zeng, Zhen Xiang, Yue Xing, Jiliang Tang, and Pengfei He. 2025. Unveiling privacy risks in LLM agent memory. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25241–25260, Vienna, Austria. Association…
  14. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091. find
  15. Qingyun Wu and 1 others. 2024. Autogen: Enabling next-gen llm applications via multi-agent conversation. In International Conference on Learning Representations (ICLR).
  16. Shijie Wu, Ozan Irsoy, and 1 others. 2023. Bloomberggpt: A large language model for finance. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  17. Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, and 15 others. 2024. The finben: An holistic financial b… — [arXiv:2402.12659](https://arxiv.org/abs/2402.12659) find
  18. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. Pixiu: A large language model, instruction data and evaluation benchmark for finance. Preprint, arXiv:2306.05443. — [arXiv:2306.05443](https://arxiv.org/abs/2306.05443) find
  19. Shuo Yang, Yilun Liu, and 1 others. 2023. Fingpt: Open-source financial large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Industry Track.
  20. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.
  21. Shunyu Yao, Jeffrey Zhao, and 1 others. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  22. Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. 2024. Cut the crap: An economical communication pipeline for llm-based multi-agent systems. arXiv:2410.02506. — [arXiv:2410.02506](https://arxiv.org/abs/2410.02506) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-rethinking-scale-deployment-trade-offs/