Elamur Research⌕
☰
home / papers / 2026 / TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification

TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification

Summary

The authors tackle AlexandriaX-2026 Subtask 3: given an English sentence and its machine translation into an Arabic dialect, find every erroneous character span and assign it one of six error categories. The task covers five English-to-dialect directions. The official baseline asks a 3B generative LLM (NileChat) to emit spans as JSON. The authors argue this produces broken outputs and unreliable offsets, and that it ignores the signal from the error-free majority of tokens. They instead treat the problem as discriminative token classification over the untouched MT output. Sub-word offsets are kept so that predicted labels map back exactly to character positions. The training data are tiny (1,125 sentences) and heavily skewed, so they add three measures: a focal loss that down-weights the no-error label, confidence thresholds tuned separately for each dialect, and merging of the rarest categories. They compare six Arabic encoders, and MARBERTv2 wins with 40.8 on dev and 40.91 on test, placing 3rd. A later run without category merging reached 42.00. Locating errors works reasonably well, but rare categories remain essentially unlearned.

Key points

  • Data:
    • 1,125 training sentences containing 1,997 error spans (1.78 per sentence), plus a 138-sentence dev set.
    • About 80% of tokens carry no error.
    • Sociolinguistics accounts for 57.9% of spans, while graphetics has only 2 spans.
  • Loss and category handling:
    • Focal loss uses γ = 2, α_O = 0.3 and α = 1 for the error classes.
    • Orthography, pragmatics and graphetics were merged into a single "other" class at training time.
    • The submitted system kept this merge. A resubmission without it raised the test Overall Score from 40.91 to 42.00.
  • Decoding gains (out-of-fold): per-dialect thresholds added up to 0.8 points, and span trimming plus a minimum-length filter added 0.3–0.5 points.
  • Encoders compared: MARBERTv2, a QuranQA-tuned MARBERTv2 checkpoint, CAMeLBERT-DA, SaudiBERT, AraBERTv2 and NileChat-3B used as a tagger. MARBERTv2 clearly led.
  • Ensemble and per-category results:
    • A character-level voting ensemble scored 39.9 on dev, below the single best model's 40.8.
    • Per-category OOF F1 was 39.2 for sociolinguistics, 23.6 for semantics and only 5.3 for morphosyntax.
    • Pragmatics and graphetics stayed at 0.

Interesting ideas

  • I1. Labelling the raw, unnormalized string while carrying each sub-word's character offsets makes span output exact by construction; normalization would silently misalign it. — For Mova (tokens): The en tokenizer and tagger could keep original character offsets through every normalization step, so each lemma or tag can be traced back to its exact source substring when explaining a decision.
  • I2. A focal loss with a reduced weight on the dominant "nothing here" label keeps a small, skewed dataset from collapsing into always predicting the majority class. — For Mova (learn): When training classifiers for rare dependency relations or rare repair-rule triggers, down-weighting the majority label could keep tail cases from being drowned out.
  • I3. Decision thresholds are tuned per subpopulation (here, per dialect) on out-of-fold predictions, because error density differs between groups and one global cutoff misfires in opposite directions. — For Mova (learn): Silver-data acceptance thresholds could be calibrated per genre or source (school texts, fables, word problems) on held-out folds rather than set once globally.
  • I4. Character-level voting with an adjustable minimum number of agreeing models slides between union (recall) and intersection (precision). Checking it against the best single model showed that weaker, correlated voters dilute rather than help. — For Mova (learn): Agreement among witness parsers could be tuned the same way, and should always be compared with the strongest single parser, since adding correlated weak witnesses may lower silver quality.
  • I5. The authors report both pooled and nested out-of-fold scores, where nested tunes thresholds only on the other folds. They also ran the per-category analysis on training OOF predictions because the dev set lacked the rare classes entirely. — For Mova (learn): Evaluation reports could pair each tuned score with a nested, leakage-free variant, and check that test splits actually contain the rare phenomena being measured.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Part-of-speech tagging · Machine translation

Cited by

1 paper of our library cites it (of the 140884 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2609.29633 · arXiv 2609.29633

@misc{abusaleh2026ttlab,
  title = {{TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification}},
  author = {Abusaleh, Ali and Verma, Bhuvanesh and Mehler, Alexander},
  year = {2026},
  doi = {10.48550/arxiv.2609.29633},
  eprint = {2609.29633},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.29633}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗ our card
  2. Abdullah Alabdullah, Lifeng Han, and Chenghua Lin. 2025. Advancing dialectal arabic to modern standard arabic machine translation. Preprint, arXiv:2507.20301. ↗ our card
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resources Association. find our card
  4. Abdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata, and Muhammad Abdul-Mageed. 2025. NileChat: Towards linguistically diverse and culturally aware LLMs for local communities. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10978–11002, Suzhou, China. Association for Computational Linguistics. find
  5. Abdellah El Mekki, AbdelRahim A. Elmadany, Samar M. Magdy, Saad Ezzini, Mo El-Haj, Mustafa Jarrar, Zaid Alyafeai, Bernard Ghanem, and Muhammad Abdul-Mageed. 2026. AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation. In Proceedings of The Fourth Arabic Natural Language Processing Conference: Shared Tasks, Budapest, Hungary. Association for Computational Linguistics. find our card
  6. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. find our card
  7. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xCOMET: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. find our card
  8. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find our card
  9. Dahyun Jung, Chanjun Park, Sugyeong Eo, and Heuiseok Lim. 2023. Enhancing machine translation quality estimation via fine-grained error analysis and large language model. Mathematics, 11(19). find
  10. Tom Kocmi and Christian Federmann. 2023. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768–775, Singapore. Association for Computational Linguistics. find our card
  11. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2018. Focal loss for dense object detection. Preprint, arXiv:1708.02002. ↗ our card
  12. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumàtica tecnologies de la traducció, pages 455–463. find
  13. Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki, Wesam El Sayed, and Muhammad Abdul-Mageed. 2026. LQM: Linguistically motivated multidimensional quality metrics for machine translation. In Findings of the Association for Computational Linguistics: ACL 2026, pages 40470–40493, San Diego, California, United States. Association for Computational Linguistics. find
  14. Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR – grammatical error correction: Tag, not rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163–170, Seattle, WA, USA → Online. Association for Computational Linguistics. find
  15. Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Niccolò Campolungo, and Roberto Navigli. 2022. MaTESe: Machine translation evaluation as a sequence tagging problem. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 569–577, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  16. Faisal Qarah. 2024. Saudibert: A large language model pretrained on saudi dialect corpora. arXiv preprint arXiv:2405.06239. ↗ our card
  17. WMT QE Shared Task Organizers. 2023. WMT 2023 shared task on quality estimation — task 2: Fine-grained error span detection. https://wmt-qe-task.github.io/ wmt-qe-2023/subtasks/task2/. Accessed 9th, July 2026. ↗
  18. Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orăsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 shared task on quality estimation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 69–99, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abusaleh-2026-ttlab-alexandriax-2026-fine-tuned/