Elamur Research⌕
☰
home / papers / 2026 / ASCAT: Arabic Scientific Benchmark for Advanced Translation Evaluation

ASCAT: Arabic Scientific Benchmark for Advanced Translation Evaluation

Summary

Arabic lacks parallel English–Arabic resources for scientific machine translation. Existing corpora are either general-domain or made of short units such as thesis titles or single sentences. ASCAT fills this gap with a small but carefully validated evaluation benchmark of 500 complete arXiv abstracts, sampled from the gfissore/arxiv-abstracts-2021 dataset. Their subject matter spans artificial intelligence, computer science and mathematics, along with physics and its quantum mechanics subfield.

Each abstract was first machine-translated by three different systems: Gemini, the Hugging Face quickmt-en-ar model and Google Translate. Seven domain experts then post-edited the output against a checklist covering terminology, named entities, grammar, structure, hedging and consistency. LaTeX and symbols were left untranslated. No inter-annotator agreement was measured.

The authors tested three LLMs against the expert references. BLEU scores differ by roughly 13 points between the best and worst models, which the authors take as evidence that the benchmark separates systems well. The resource matters because it supports document-level evaluation of Arabic scientific translation, where terminology and multi-sentence discourse are central.

Key points

  • Size and length: 500 abstracts averaging 125.3 English words and 111.78 Arabic words. The corpus totals 67,293 English and 60,026 Arabic tokens.

  • Vocabulary: Arabic has 17,604 unique words versus 12,685 in English (TTR 0.29 vs. 0.19), which the authors attribute to Arabic morphology. Sentence counts per abstract are nearly equal (7.16 vs. 6.99).

  • Benchmark results:

    ModelBLEUROUGE-L
    GPT-4o-mini37.070.586
    Gemini-3.0-Flash-Preview30.440.522
    Qwen3-235B-A22B23.680.531

    Qwen slightly outperforms Gemini on ROUGE-1 and ROUGE-2.

  • Post-editing: edits were classified as substitutions, insertions, deletions or rewrites. Experts spent about 10–30 minutes per sample.

  • Limitations: the corpus is small and its domain distribution is uneven. Evaluation relies only on automatic metrics, and there is no human evaluation or IAA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3328e5bk8rrs

@inproceedings{sibaee2026ascat,
  title = {{ASCAT: Arabic Scientific Benchmark for Advanced Translation Evaluation}},
  author = {Sibaee, Serry and Jallad, Khloud Al and Yousfi, Zineb and Elhosiny, Israa and El-Ghawi, Yousra Yousra and Balah, Batool and Nacar, Omer},
  booktitle = {The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks},
  year = {2026},
  doi = {10.63317/3328e5bk8rrs},
  url = {https://doi.org/10.63317/3328e5bk8rrs}
}

References

8 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rania Al-Sabbagh. 2024. Peach: A sentence-aligned parallel english-arabic corpus for healthcare. Corpora, 19(3):395–410. ↗
  2. M. A. Author and Others. 2026. Deast: A dataset for english-arabic scientific translation and vice versa. Data in Brief, 64:112381. find
  3. Andreas Eisele and Yu Chen. 2010. MultiUN: A multilingual corpus from united nation documents. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). find our card
  4. K. Hennara, M. Hreden, M. M. Hamed, Z. Aldallal, S. Chrouf, and S. AlModhayan. 2025. Mutarjim: Advancing bidirectional arabic-english translation with a small language model. arXiv preprint arXiv:2505.17894. ↗
  5. Mahmoud S. Mohammed and Mohamed Khalil. 2025. Athar: A high-quality and diverse dataset for classical arabic to english translation. In Proceedings of the Third Arabic Natural Language Processing Conference, pages 97–106. ↗
  6. Jack C Richards and Richard W Schmidt. 2013. Longman dictionary of language teaching and applied linguistics. Routledge. ↗
  7. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of LREC 2012, pages 2214–2218. find
  8. Jianhao Yan, Pingchuan Yan, Yulong Chen, Judy Li, Xianchao Zhu, and Yue Zhang. 2024. Gpt-4 vs. human translators: A comprehensive evaluation of translation quality across languages, domains, and expertise levels. ↗

This page in Markdown → https://md.research.elamur.ai/free/papers/sibaee-2026-ascat-arabic-scientific-benchmark-advanced/