Elamur Research⌕
☰
home / papers / 2026 / Arabic ChartSumm: An English-to-Arabic Benchmark for Metadata-to-Text Summarization

Arabic ChartSumm: An English-to-Arabic Benchmark for Metadata-to-Text Summarization

Paper · 2026 · Passant Elchafei and Amany Fashwan

Summary

Arabic has had no benchmark for turning chart data into prose summaries, while English has several. The authors build Arabic ChartSumm by machine-translating the Statista portion of the English ChartSumm benchmark into Modern Standard Arabic. They use two translators with different designs, NLLB-200-distilled-600M and Qwen2.5-1.5B, and keep numeric values untouched. Translation quality is measured with BLEU, chrF, COMET_ref and COMET_QE against a Google Translate pivot, plus a manually reviewed set of 1,000 test pairs. NLLB clearly wins on both surface and semantic fidelity.

Each translated version is then used to fine-tune two summarizers, the multilingual mT5 and the Arabic-specific AraBART. Their inputs are chart metadata (title, axis labels, data series) flattened into structured text prompts. The best pairing is AraBART trained on NLLB data, reaching ROUGE-L 63.8 and BLEU 33.1 on the test set. Models trained on Qwen translations fall well behind. The work provides a first Arabic data-to-text resource and shows that downstream quality depends strongly on how good the translation is.

Key points

  • The corpus has 41,184 Arabic chart–summary pairs from ChartSumm's Statista subset, split into 32,985 train, 4,101 validation and 4,098 test pairs, matching the original splits.
  • Each instance is a JSON record with a title, x_label/y_label, a data table mapping categories to numbers, a human-written summary and a link to the chart image. Text fields are translated and numbers are left as they are.
  • On the test set, NLLB+AraBART scores ROUGE-L 63.8 and BLEU 33.1. NLLB+mT5 scores 63.2 and 21.5, Qwen+mT5 scores 57.8 and 17.3, and Qwen+AraBART collapses to 22.6 and 13.0.
  • A pivot-based evaluation keeps the same ranking. NLLB+AraBART leads with ROUGE-L 37.5, BLEURT 23.75, CIDEr 86.24 and Content Selection 86.33.
  • Only the NLLB translation is released, on GitHub (a-fashwan/Arabic-ChartSumm-Dataset). Stated limitations include translated rather than natively written references, possible stylistic bias from the Google pivot, and no human evaluation of the summaries.

Interesting ideas

  • I1. The authors treat translation quality as a controlled variable: two translators feed identical downstream training, so differences in summary quality can be traced back to translation fidelity.
  • I2. A reference-based score against a pivot is paired with a reference-free quality estimate (COMET_QE), which partly offsets bias toward the pivot's own style.
  • I3. Numeric values are frozen during translation, and only labels and prose are converted, which protects the facts that matter most in a chart.
  • I4. Structured chart metadata (title, axes, series) is flattened into a templated natural-language prompt before generation. This gives a plain-text model an ordered, labelled view of the data.
  • I5. A Content Selection metric scores whether a summary mentions the right data items, separately from n-gram overlap.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when building silver data through different teachers or parsers, train the same downstream reader on each version to measure how much the source of the silver data actually matters.
  • I2 (learn): evaluation of parses or frames could combine agreement with one reference parser with a reference-free plausibility check, so the system is not just rewarded for copying one witness's dialect.
  • I3 (back): when translating or compressing text through the meaning graph, quantities and measures should be copied as locked values, not regenerated, so the round trip cannot corrupt numbers.
  • I4 (back): world-state entities and holdings could be serialized in a fixed, labelled order before surface realization, making generation traceable back to each graph node.
  • I5 (answer): answers to word problems or fable QA could be checked for whether they mention the correct entities and values, not only for how closely they match the gold wording.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 4 English · 1,373 works Evaluation & benchmarks · 685 works Semitic languages · 1,484 works Summarization · 159 works

Cite

Cite as Passant Elchafei and Amany Fashwan. 2026. Arabic ChartSumm: An English-to-Arabic Benchmark for Metadata-to-Text Summarization. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5pxwxs9vtum4

@inproceedings{elchafei2026arabic,
  title = {{Arabic ChartSumm: An English-to-Arabic Benchmark for Metadata-to-Text Summarization}},
  author = {Passant Elchafei and Amany Fashwan},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5pxwxs9vtum4},
  url = {https://doi.org/10.63317/5pxwxs9vtum4}
}

References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohamed Yassin Abdelwahab, Yazeed Al Moaiad, and Zainab Binti Abu Bakar. 2023. Arabic text summarization using pre-processing methodologies and techniques. Asia-Pacific Journal of Information Technology & Multimedia, 12(1). ↗
  2. Molham Al-Maleh and Said Desouki. 2020. Arabic text summarization using deep learning approach. Journal of Big Data, 7(1):109. ↗
  3. Lamees Mahmoud Al Qassem, Di Wang, Zaid Al Mahmoud, Hassan Barada, Ahmad Al-Rubaie, and Nawaf I Almoosa. 2017. Automatic arabic summarization: a survey of methodologies and systems. Procedia Computer Science, 117:10– 18. find
  4. Sultan Alrashed, Dmitrii Khizbullin, and David R. Pugh. 2024. FineWeb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Models. Technical Report arXiv:2411.06402, King Abdullah University of Science and Technology (KAUST). CC-BY-NC-4.0 licensed; contains 202B tokens. ↗
  5. Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022. Arabart: a pretrained arabic sequence-to-sequence model for abstractive summarization. arXiv preprint arXiv:2203.10945. ↗
  6. Khalid N Elmadani, Mukhtar Elgezouli, and Anas Showk. 2020. Bert fine-tuning for arabic text summarization. arXiv preprint arXiv:2004.14135. ↗ our card
  7. Nada Essa, MM El-Gayar, and Eman M El-Daydamony. 2025. Enhanced model for abstractive arabic text summarization using natural lan-ROUGE-2 20.43 24.26 16.57 08.69 ROUGE-L 34.96 37.50 31.41 14.95 ROUGE-Lsum 34.95 37.54 31.40 14.93 BLEURT 20.83 23.75 16.85 10.48 CIDEr 56.97 86.24 35.76 17.67 CS 83.28 86.33 73.42 41.24 PPL 63.47 85.13 56.71 20.16 guage generation and named entity recognition. Neural Computing and Applications, pages 1–23. find
  8. Wael Etaiwi and Arafat Awajan. 2022. Semg-ts: Abstractive arabic text summarization using semantic graph embedding. Mathematics, 10(18):3225. ↗
  9. Enamul Hoque and M Saidul Islam. 2025. Natural language generation for visualizations: State of the art, challenges and future directions. In Computer Graphics Forum, volume 44, page e15266. Wiley Online Library. ↗
  10. Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022. AraBART: a pretrained Arabic sequence-to-sequence model for abstractive summarization. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 31–42, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  11. Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale benchmark for chart summarization. arXiv preprint arXiv:2203.06486. ↗ our card
  12. Mengsha Liu, Daoyuan Chen, Yaliang Li, Guian Fang, and Ying Shen. 2024. Chartthinker: A contextual chain-of-thought approach to optimized chart summarization. arXiv preprint arXiv:2403.11236. ↗
  13. Malak Mashaabi, Shahad Al-Khalifa, and Hend Al-Khalifa. 2024. A survey of large language models for arabic language and its dialects. arXiv preprint arXiv:2410.20238. ↗
  14. Sari Masri, Yaqeen Raddad, Fidaa Khandaqji, Huthaifa I. Ashqar, and Mohammed Elhenawy. 2025. Transformer models in education: Summarizing science textbooks with arabart, mt5, arat5, and mbart. In Intelligent Systems, Blockchain, and Communication Technologies, volume 1268 of Lecture Notes in Networks and Systems, pages 286–300. Springer, Cham. ↗ our card
  15. Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. 2023. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761. ↗ our card
  16. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics. our card
  17. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. Describes the NLLB-200 model family, including the distilled 600M variant. ↗
  18. Jason Obeid and Enamul Hoque. 2020. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. arXiv preprint arXiv:2010.09142. ↗
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics. our card
  20. Maja Popovic. 2015. chrf: character n-gram f-score for automatic mt evaluation. In WMT@EMNLP. our card
  21. Raian Rahman, Rizvi Hasan, Abdullah Al Farhad, Md Tahmid Rahman Laskar, Md Hamjajul Ashmafee, and Abu Raihan Mostofa Kamal. 2023. find
  22. Chartsumm: A comprehensive benchmark for automatic chart summarization of long and short summaries. arXiv preprint arXiv:2304.13620. Version v3, last revised 11 Jun 2023. ↗
  23. Ricardo Rei, Craig Alan Stewart, Ana C. Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. ArXiv, abs/2009.09025. ↗ our card
  24. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. ↗ our card
  25. Hao Tan, Chen-Tse Tsai, Yujie He, and Mohit Bansal. 2022. Scientific chart summarization: Datasets and improved text modeling. In SDU@ AAAI. find
  26. Peixin Xu, Yujuan Ding, and Wenqi Fan. 2024. Chartadapter: Large vision-language model for chart summarization. arXiv preprint arXiv:2412.20715. ↗
  27. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. CoRR, abs/2010.11934. ↗ our card
  28. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 483– 498. ↗ our card
  29. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Describes the Qwen2.5 series including the 1.5B variant. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elchafei-2026-arabic-chartsumm-english-arabic-benchmark/