Elamur Research⌕
☰
home / papers / 2024 / From Nile Sands to Digital Hands: Machine Translation of Coptic Texts

From Nile Sands to Digital Hands: Machine Translation of Coptic Texts

Paper · 2024 · Muhammed Saeed, Asim Mohamed, Mukhtar Mohamed et al.

Summary

Coptic, the late stage of the Egyptian language, is still used in church liturgy, but it has almost no digital resources and current language technology largely ignores it. The authors collected Coptic material available online and aligned it into a parallel corpus. In the abstract they report roughly 8,000 Arabic–Coptic and over 24,000 English–Coptic sentence pairs. On this corpus they trained a Seq2Seq transformer from scratch, using both word-level and BPE tokenization. They also fine-tuned mT5 and M2M100, each on separate language pairs and as a single many-to-many model. Separately, they tested GPT-4, Claude 3 Opus, Gemini 1.5 Pro and Llama-3-70b in zero-shot and five-shot setups. mT5 has no Coptic characters in its vocabulary, yet it learned to produce Coptic output. M2M100 failed and mostly emitted unknown tokens. Among the LLMs, Claude 3 Opus scored highest, and five examples in the prompt improved results substantially. The work supplies the first neural MT resources for Coptic and indicates that some pretrained models can adapt to scripts they have never seen.

Key points

  • The corpus, code and models are released at github.com/UBC-NLP/copticmt. The Arabic–Coptic size is given inconsistently in the text (about 8,000 in the abstract, 16,800 in the introduction), while English–Coptic is about 24,000 pairs.
  • The baseline is a transformer with 6 encoder layers, 6 decoder layers and 8 attention heads, tested with word-level and BPE tokenization. The word-level version reached 7.41 BLEU for Coptic→English.
  • The best supervised result came from pairwise mT5 trained on English–Coptic: 14.9 BLEU for Coptic→English and 6.84 for English→Coptic. On these pairs the many-to-many mT5 scored lower than the pairwise models.
  • M2M100 could not encode or decode Coptic and produced frequent <unk> tokens, so the authors did not report its scores.
  • The LLMs translated out of Coptic better than into it. Claude 3 Opus led in most settings, and five-shot prompting helped every model. Arabic–English was included as a reference pair for languages the models saw in pretraining.

Interesting ideas

  • I1. A high-resource pair (Arabic–English) was evaluated alongside the low-resource pairs, which shows how far each model falls below its own ceiling instead of reporting raw scores in isolation.
  • I2. Scoring each direction separately exposed a gap between understanding Coptic and producing it, which a single averaged number would hide.
  • I3. Pooling all languages into one many-to-many model did worse than dedicated pairwise models on this scarce data, so sharing a model is not automatically beneficial.
  • I4. A handful of in-prompt examples greatly lifted LLM quality on a nearly unseen language.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • Before choosing a model, the authors checked whether its vocabulary covers the target script at all, and this check predicted why M2M100 collapsed into <unk> while mT5 coped. — (tokens): Add a coverage audit that reports which characters or word shapes the tokenizer and lexicon cannot represent, so that silent <unk>-like failures are caught before tagging.
  • I1 (learn): Pair every evaluation on a hard domain with a scored control domain the system already handles well, so that drops can be attributed to the domain and not to the pipeline.
  • I2 (back): Evaluate text→graph and graph→text independently in the compressor, so that weak generation is not hidden behind strong parsing or the reverse.
  • I3 (learn): When mixing silver data from different treebanks or genres, compare against per-source training before assuming that more pooled data helps.
  • I4 (learn): Include a few verified, in-domain examples when querying LLM teacher tiers for silver annotations, especially for rare constructions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksSemitic languages · 2,908 works

Cite

Cite as Muhammed Saeed, Asim Mohamed, Mukhtar Mohamed, Shady Shehata and Muhammad Abdul-Mageed. 2024. From Nile Sands to Digital Hands: Machine Translation of Coptic Texts. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.25

@inproceedings{saeed2024nile,
  title = {{From Nile Sands to Digital Hands: Machine Translation of Coptic Texts}},
  author = {Muhammed Saeed, Asim Mohamed, Mukhtar Mohamed, Shady Shehata and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.25},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.25}
}

References

29 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Orevaoghene Ahia and Kelechi Ogueji. 2020. Towards supervised and unsupervised neural machine translation baselines for nigerian pidgin. arXiv preprint arXiv:2003.12660. ↗
  2. Anthropic. 2024. Claude 3 haiku: Our fastest model yet. https://www.anthropic.com/ news/claude-3-haiku. ↗
  3. Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Mitiku Yohannes Fuge, Aman Kassahun Wassie, Eyasu Shiferaw Jada, Yonas Chanie, Walelign Tewabe Sewunetie, and Seid Muhie Yimam. 2024. Walia-llm: Enhancing amharic-llama by integrating task-specific and generative datasets. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario https://alliancecan.ca ↗
  4. Amodei. 2020. Language models are few-shot learners.
  5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. ↗
  6. Wikipedia contributors. 2024. Coptic diaspora. Wikipedia, The Free Encyclopedia. Accessed: 2024-06-21. find
  7. Yiming Cui, Ziqing Yang, and Xin Yao. 2024. Efficient and effective text encoding for chinese llama and alpaca. ↗
  8. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22(1). find
  9. Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. 2023. Dictionary-based phrase-level prompting of large language models for machine translation. arXiv preprint arXiv:2302.07856. ↗
  10. Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023. Continual pre-training of large language models: How to (re)warm your model? ↗
  11. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. find
  12. Jaavid Aktar Husain, Raj Dabre, Aswanth Kumar, Jay Gala, Thanmay Jayakumar, Ratish Puduppully, and Anoop Kunchukuttan. 2024. Romansetu: Efficiently unlocking multilingual capabilities of large language models models via romanization. find
  13. Pin-Jie Lin, Muhammed Saeed, Ernie Chang, and Merel Scholman. 2023. Low-resource cross-lingual adaptive training for nigerian pidgin. In Proceedings of the 24th INTERSPEECH conference.
  14. Inc OpenAI. 2024. Gpt4. https://platform.openai.com/docs/models/ gpt-4-turbo-and-gpt-4 [Accessed: (Use the date of access)]. ↗
  15. Asim Awad Osman, Ahmed Emadeldin Almahady, Muhammed Saeed, and Hiba Hassan Sayed. 2023. Machine translation baselines for arabic-swahili. find
  16. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. ↗
  17. Suzanna Sia and Kevin Duh. 2022. Prefix embeddings for in-context machine translation. In Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 45–57. find
  18. Suscopts. n.d. Coptic alphabet. PDF document, Suscopts. Accessed: 2024-06-21. find
  19. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
  20. The Editors of Encyclopaedia Britannica. 2017. Suhaj governorate, egypt. Encyclopedia Britannica. Accessed: 2024-06-21. find
  21. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  23. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen tse Huang, Wenxiang Jiao, and Michael R. Lyu. 2023. All languages matter: On the multilingual safety of large language models. find
  24. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. ↗
  25. Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. Prompting large language model for machine translation: A case study. ↗
  26. Chen Zhang, Xiao Liu, Jiuheng Lin, and Yansong Feng. 2024. Teaching large language models an unseen language on the fly. arXiv preprint arXiv:2402.19167. ↗
  27. Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023b. Don’t trust ChatGPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7915–7927, Singapore. Association for Computational Linguistics. find
  28. Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. ↗
  29. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual machine translation with large language models: Empirical results and analysis.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saeed-2024-nile-sands-digital-hands-machine/