📄 ASCAT: An Arabic Scientific Corpus and Benchmark for Advanced Translation Evaluation

Summary

ASCAT addresses the shortage of carefully checked English–Arabic parallel data for scientific translation. Existing resources tend to use short segments such as titles or leaflet sentences, or they cover a single domain. The authors collected 500 full scientific abstracts from physics, mathematics, computer science, quantum mechanics and AI. Each abstract was machine-translated by several kinds of systems: Gemini, the Hugging Face model quickmt-en-ar, and the commercial services Google Translate and DeepL. Seven experts with graduate degrees then reviewed the output independently, working only within their own field. They followed a checklist covering terminology, named entities, grammar, structure, hedging and sentence count, and they resolved disagreements by reaching consensus.

The resource is meant as an evaluation benchmark, not a training set. Three LLMs were scored against the validated references with BLEU and ROUGE. GPT-4o-mini led, Gemini-3.0-Flash-Preview came second and Qwen3-235B-A22B came last, with a spread of about 13 BLEU points. The authors read this spread as evidence that the benchmark can separate systems. Even the best score is moderate, which shows that long-form scientific Arabic remains hard to translate.

Key points

• The corpus has 500 abstracts with 67,293 English and 60,026 Arabic tokens. There are 12,685 unique English words and 17,604 unique Arabic words, giving a TTR of 0.19 for English and 0.29 for Arabic.
• The reported mean length is inconsistent. Table 3 gives 125.3 words for English and 111.8 for Arabic, while the abstract states 141.7 words for English. Sentence counts per abstract are nearly equal at 7.16 and 6.99.
• BLEU and ROUGE-L scores:
  • GPT-4o-mini: BLEU 37.07, ROUGE-L 0.586.
  • Gemini-3.0-Flash-Preview: BLEU 30.44, ROUGE-L 0.522.
  • Qwen3-235B-A22B: BLEU 23.68, ROUGE-L 0.531.
• Table 1 compares ASCAT with DEAST, PEACH, ATHAR and Tarjama-25. ASCAT is the smallest of these but has the longest units and multi-stage validation.
• The authors list several limitations:
  • The domains are unevenly distributed.
  • Evaluation uses only automatic metrics.
  • Some phenomena remain unsolved: terms whose meaning shifts between fields (e.g. "state"), unstandardized terms, hedging, and eponyms such as Wasserstein distance.

Interesting ideas

• I1. Several dissimilar translation engines produce candidates for each text, and experts review them with the alternatives side by side, which lets them focus on the risky spots instead of editing a single draft blindly. — For Mova (learn): Silver-data agreement between witness parsers could likewise route only contested arcs or frames to the LLM teacher or a human, while agreed parts pass with less scrutiny.
• I2. The validation checklist is organised by linguistic level (lexical, syntactic, semantic, discourse) and uses yes/no criteria, which makes each judgement auditable. — For Mova (back): Generated or translated text from the meta-world could be checked level by level against such a checklist, giving an explainable pass/fail per layer instead of one opaque score.
• I3. Matching sentence counts between source and target serves as a cheap structural fidelity check that catches merged or split sentences. — For Mova (back): The text → graph → text compressor could assert that the regenerated text keeps sentence and event counts, which would serve as a negative control for dropped or fused propositions.
• I4. Preservation of epistemic hedging ("may suggest", "appears to") is treated as an explicit quality criterion. — For Mova (frames): Prag could mark hedged speech acts as a modality feature on frames, so the text world stores claims as tentative and the answer layer does not overstate them.
• I5. When BLEU and ROUGE disagree, the gap is read as a diagnostic: the output has good content coverage but paraphrased structure, or the right words in a distant structure. — For Mova (learn): Evaluation could pair a strict structural metric with a lexical-coverage metric so their gap shows whether a regression comes from wrong content or merely different phrasing.