Elamur Research⌕
☰
home / papers / 2026 / Data Contamination in Neural Hieroglyphic Translation: A Reproducibility Study

Data Contamination in Neural Hieroglyphic Translation: A Reproducibility Study

Paper · 2026 · Ammar Toutou, Abdelrahman Harb and Christine Basta

Summary

De Cao et al. (2024) reported 61.5 BLEU for translating Ancient Egyptian hieroglyphs (in Gardiner notation) into German with a fine-tuned M2M-100. That is a striking figure for a corpus that offers only 18,669 usable TLA training pairs. Toutou, Harb and Basta tried to reproduce it, but the public checkpoint reached just 37.0 BLEU. They then audited the hiero-transformer data. After normalization, about one third of the 50 test sentences have German references that also occur in training. The authors trace this to random sentence-level splitting of texts full of recurring formulae, such as offering texts, medical recipes and royal titles. They scored five systems separately on contaminated and clean items with BLEU, chrF++ and COMET-22: the released and retrained M2M-100 variants plus mBART-50. They also measured character 8-gram overlap and document-level leakage. Every model does far better on memorized references than on unseen ones. The authors release a 34-item clean test set, detection scripts and realistic baselines. They argue that every ancient-language MT benchmark should be checked for target overlap before its scores are trusted.

Key points

  • Exact normalized matching flags 16 of 50 test targets (32%). A looser test, 8-gram overlap at a 70% threshold, flags 25 of 50 (50%). The validation set shows a similar exact rate of 30.7%.
  • On contaminated items the best model reaches 83.8 BLEU / 0.924 COMET-22. On clean items the five configurations score 30.9–39.2 BLEU / 0.622–0.676 COMET-22, a gap of 29–47 BLEU across both M2M-100 and mBART-50.
  • Several references were memorized independently of the input. In 10 of the 16 contaminated cases, a matching training sentence comes from an almost unrelated inscription, with source similarity below 5%.
  • Removing leaking documents lowers contaminated-subset BLEU by only 4.6 points, because 8 of the 16 targets also appear in other documents. The authors conclude that deduplication must happen at the target level.
  • The released resources include the decontaminated 34-sample test set, a per-item contamination catalogue, scripts and checkpoints (github.com/ammarlhassan/hiero-contamination-study). The authors also propose a document-level, genre-stratified splitting protocol.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of the Basque CountryLeioa · Spain57 works · 16,520 citations🎓Alexandria UniversityAlexandria · Egypt39 works · 287 citations

Also on the byline: AlAlamein International University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Ammar Toutou, Abdelrahman Harb and Christine Basta. 2026. Data Contamination in Neural Hieroglyphic Translation: A Reproducibility Study. Proceedings of the 6th International Conference on Natural Language Processing for the Digital Humanities.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.nlp4dh-1.6

@inproceedings{toutou2026data,
  title = {{Data Contamination in Neural Hieroglyphic Translation: A Reproducibility Study}},
  author = {Ammar Toutou, Abdelrahman Harb and Christine Basta},
  booktitle = {Proceedings of the 6th International Conference on Natural Language Processing for the Digital Humanities},
  year = {2026},
  doi = {10.18653/v1/2026.nlp4dh-1.6},
  url = {https://doi.org/10.18653/v1/2026.nlp4dh-1.6}
}

References

8 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Resear… find
  2. Mattia De Cao, Nicola De Cao, Angelo Colonna, and Alessandro Lenci. 2024. Deep learning meets egyptology: A hieroglyphic transformer for translating Martin Popel, Marketa Tomkova, Jakub Tomek, Łukasz Kaiser, Jakob Uszkoreit, Ondřej Bojar, and Zdeněk Žabokrtský. 2020. Transforming machine translati… find
  3. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395.
  4. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191.
  5. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages…
  6. Thea Sommerschield, Yannis Assael, John Pavlopoulos, Vanessa Stefanak, Andrew Senior, Chris Dyer, John Bodel, Jonathan Prag, Ion Androutsopoulos, and Nando de Freitas. 2023. Machine learning for ancient languages: A survey. Computational Linguistics, 49(3):703–747.
  7. David Tan, Pinzhen Chen, Josef van Genabith, and Koel Dutta Chowdhury. 2026. When flores bloomz wrong: Cross-direction contamination in machine translation evaluation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026).
  8. Philipp Wiesenbach and Stefan Riezler. 2019. Multi-task modeling of phonographic languages: Translating middle Egyptian hieroglyphs. In Proceedings of the 16th International Conference on Spoken Language Translation. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/toutou-2026-data-contamination-neural-hieroglyphic-translation/