Elamur Research⌕
☰
home / papers / 2026 / BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization

BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization

Paper · 2026 · Ahmed Rafid, Rumman Adib, Fariya Ahmed et al.

Summary

Automatic checks of whether a summary stays faithful to its source are largely unavailable for Bangla. Existing tools such as QuestEval, SummaC, FactScore and QAFactEval do not support the language, and n-gram metrics like ROUGE and BLEU struggle with its rich morphology and flexible word order. The authors adapt QuestEval's reference-free, question-answering approach into BanglaSummEval.

The pipeline extracts named entities and nouns as candidate answers and generates questions about them from both the source and the summary. It answers those questions against the other text, keeps only questions that pass a self-consistency check, and weights each question by importance. All four roles are handled by one 4-bit quantized Qwen3-14B model. Answers are compared with BERTScore-Recall rather than token F1.

On 300 human-written educational and medical summaries, the scores correlate well with expert ratings. The framework also exposes intermediate steps, so errors can be traced to a specific stage. It runs on free cloud GPUs, which makes it practical for low-resource research settings.

Key points

  • Backbone selection: BanglaT5, mT5-Base, Qwen3-8B-bnb-4bit and Qwen3-14B-bnb-4bit were compared on 150 BanglaRQA passages (300 QA and 300 QG pairs). Qwen3-14B-bnb-4bit performed best, with BERTScore-F1 of 0.67 for QA and 0.79 for QG.
  • Agreement with experts: on 300 summaries, correlation with human judgments was Pearson's r = 0.694, Spearman's ρ = 0.763 and Kendall's τ = 0.644, with R² = 0.481.
  • Setup: answer similarity uses BERTScore built on XLM-RoBERTa-base. A threshold τ filters out generated questions that cannot be answered reliably. The whole system fits on Kaggle or Colab free-tier GPUs.
  • Admitted limitations:
    • Overall accuracy is not much better than simply prompting an LLM to judge the summary.
    • Using one model for every stage can create self-reinforcing bias.
    • Complex relational errors, such as causal chains or role reversals, may go undetected.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓York UniversityToronto · Canada13 works · 3,075 citations🎓Islamic University of TechnologyDhaka · Bangladesh10 works · 555 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 232 works

Cite

Cite as Ahmed Rafid, Rumman Adib, Fariya Ahmed, Ajwad Abrar and Mohammed Saidul Islam. 2026. BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.51

@inproceedings{rafid2026banglasummeval,
  title = {{BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization}},
  author = {Ahmed Rafid, Rumman Adib, Fariya Ahmed, Ajwad Abrar and Mohammed Saidul Islam},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.51},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.51}
}

References

12 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati…
  2. Radia Rayan Chowdhury, Mir Tafseer Nayeem, Tahsin Tasnim Mim, Md. Saifur Rahman Chowdhury, and Taufiqul Jannat. 2021. Unsupervised abstractive summarization of Bengali text documents. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main… find
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. find
  4. Daoyuan Dai and 1 others. 2023. Gpt-ner: Named entity recognition via large language models. In Findings of the Association for Computational Linguistics: NAACL 2023.
  5. Syed Mohammed Sartaj Ekram, Adham Arik Rahman, Md. Sajid Altaf, Mohammed Saidul Islam, Mehrab Mustafy Rahman, Md Mezbaur Rahman, Md Azam Hossain, and Abu Raihan Mostofa Kamal. 2022. BanglaRQA: A benchmark dataset for under-resourced Bangla language reading comprehension-based question answering with… find
  6. Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa…
  7. Alvi Khan, Fida Kamal, Mohammad Abrar Chowdhury, Tasnim Ahmed, Md Tahmid Rahman Laskar, and Sabbir Ahmed. 2023. BanglaCHQ-summ: An abstractive summarization dataset for medical queries in Bangla conversational speech. In Proceedings of the First Workshop on Bangla Language Processing (BLP-2023), pag… find
  8. Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177. find
  9. Evaluating text generation with bert. Preprint, arXiv:1904.09675. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. P…
  10. Nishat Raihan and Marcos Zampieri. 2025. Tiger-LLM - a family of Bangla large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 887–896, Vienna, Austria. Association for Computational Linguistics. find
  11. Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021. Questeval: Summarization asks for fact-based evaluation. Preprint, arXiv:2103.12693.
  12. Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. 2023. Evaluating the factual consistency of large language models through news summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5220– 5255, Toronto, Canada. Associatio… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rafid-2026-banglasummeval-reference-free-factual-consistency/