Elamur Research⌕
☰
home / papers / 2026 / Evaluating Cost-Efficiency of LLMs in a RAG Setup on Polish Wikipedia: Quality vs. Energy Consumption

Evaluating Cost-Efficiency of LLMs in a RAG Setup on Polish Wikipedia: Quality vs. Energy Consumption

Paper · 2026 · Patrycja Smits and Tomasz Walkowiak

Summary

This student-workshop paper asks which LLM an organisation should use in a fixed Polish RAG pipeline when both answer quality and energy costs matter. The authors kept retrieval constant: a vector index over a Polish Wikipedia dump, BAAI/bge-m3 embeddings and BAAI/bge-reranker-v2-m3, with 10 passages passed to each generator. They compared seven configurations from 4B to 70B parameters: PLLuM and Bielik models, Gemma-3-4, and two GGUF-quantised variants of the largest PLLuM. Each model answered 1,000 PolQA test questions. GPT-4o judged the answers in head-to-head comparisons with bias countermeasures, and a Bradley-Terry model turned those judgments into a ranking. Energy use was measured during inference on NVIDIA H100 GPUs. Quality improved little once models exceeded about 12B parameters. PLLuM-12 performed on par with the 70B model while using roughly 83% less energy. For Polish public-sector and industry deployments, this suggests mid-size models are a better choice when retrieval supplies the knowledge.

Key points

  • Models evaluated: PLLuM-70 in FP16, Q8_0 and Q4_K_M versions, plus PLLuM-12, Bielik-11 (v2.6), Bielik-4.5 (v3.0) and Gemma-3-4. Memory needs range from 131.4 GB for PLLuM-70 down to 8.6 GB for Gemma-3-4.
  • Retrieval setup: the retriever returns the top 100 passages, the reranker keeps 10, and the prompt asks for an answer in Polish with [doc_no] citations. The authors make the dataset they produced publicly available under a CC BY-SA 4.0 licence.
  • Judging cost: five-fold self-consistency over 21 model pairs required 105,000 GPT-4o API calls.
  • Quantisation results: 8-bit quantisation kept quality close to the original and halved memory, but it raised energy use on the tested hardware. 4-bit Q4_K_M degraded quality badly and is not recommended.
  • Scope and caveats: energy figures come only from H100 GPUs with vLLM 0.10.2, and Gemma-3-4 ran inefficiently in vLLM. Answer length differed widely across models (83–670 tokens), and the findings apply specifically to Wikipedia-based RAG with dense retrieval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Wrocław University of Science and TechnologyWroclaw · Poland8 works · 945 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Patrycja Smits and Tomasz Walkowiak. 2026. Evaluating Cost-Efficiency of LLMs in a RAG Setup on Polish Wikipedia: Quality vs. Energy Consumption. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-srw.55

@inproceedings{smits2026evaluating,
  title = {{Evaluating Cost-Efficiency of LLMs in a RAG Setup on Polish Wikipedia: Quality vs. Energy Consumption}},
  author = {Patrycja Smits and Tomasz Walkowiak},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.55},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.55}
}

References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. find
  2. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Ban…
  3. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024b. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteen…
  4. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: an open platform for evaluating llms by human preference. In Proceedings of the 41st International Confere…
  5. Jae-Won Chung, Jeff J. Ma, Ruofan Wu, Jiachen Liu, Oh Jun Kweon, Yuxuan Xia, Zhiyu Wu, and Mosharaf Chowdhury. 2025. The ml.energy benchmark: Toward automated inference energy measurement and optimization. Preprint, arXiv:2505.06371. — [arXiv:2505.06371](https://arxiv.org/abs/2505.06371) find
  6. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians,…
  7. Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, and Emma Strubell. 2025. Energy considerations of large language model inference and efficiency optimizations. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  8. David R. Hunter. 2004. Mm algorithms for generalized bradley-terry models. The Annals of Statistics, 32(1):384–406. find
  9. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.
  10. Jan Kocoń and et al. 2025. Pllum: A family of polish large language models. Preprint, arXiv:2511.03823. — [arXiv:2511.03823](https://arxiv.org/abs/2511.03823) find
  11. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611…
  12. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th Internation…
  13. Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. 2023. Making large language models a better foundation for dense retrieval. Preprint, arXiv:2312.15503. — [arXiv:2312.15503](https://arxiv.org/abs/2312.15503) find
  14. Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2025. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. Preprint, arXiv:2405.04532. find
  15. Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan vulic, anna korhonen, and Nigel Collier. 2024. Aligning with human judgement: The role of pairwise preference in large language model evaluators.
  16. Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, and Adrian Gwoździej. 2025a. Bielik v3 small: Technical report. Preprint, arXiv:2505.02550. — [arXiv:2505.02550](https://arxiv.org/abs/2505.02550) find
  17. Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej, and Remigiusz Kinas. 2025b. Bielik 11b v2 technical report. Preprint, arXiv:2505.02410. — [arXiv:2505.02410](https://arxiv.org/abs/2505.02410) find
  18. GGML Organization. 2023–2025. llama.cpp: Port of transformer llms for efficient inference. https://github.com/ggml-org/llama.cpp. Accessed: 2025-12-22. ↗
  19. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. find
  20. Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. 2024. Characterizing power management opportunities for llms in the cloud. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages… find
  21. Piotr Rybak, Piotr Przybyła, and Maciej Ogrodniczuk. 2024. PolQA: Polish question answering dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12846–12855, Torino, Italia. ELRA and ICCL. find
  22. Konstantinos Vrettos and Michail E. Klontzas. 2025. Accurate and energy efficient: Local retrieval-augmented generation models outperform commercial large language models in medical tasks. Preprint, arXiv:2506.20009. — [arXiv:2506.20009](https://arxiv.org/abs/2506.20009) find
  23. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  24. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models.
  25. Konrad Wojtasik, Adrian Berdowski, Inez Okulska, and Maciej Piasecki. 2025. Polichat: Retrieval augmented generation on university documents and regulations. In Computational Science – ICCS 2025 Workshops: 25th International Conference, Singapore, Singapore, July 7–9, 2025, Proceedings, Part V, page… find
  26. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neu…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/smits-2026-evaluating-cost-efficiency-llms-rag/