Elamur Research⌕
☰
home / papers / 2023 / Multi-Parallel Corpus of North Levantine Arabic

Multi-Parallel Corpus of North Levantine Arabic

Paper · 2023 · Mateusz Krubiński, Hashem Sellat, Shadi Saleh et al.

Summary

North Levantine Arabic (Syrian/Shami, ISO code apc) is spoken widely but seldom written. Machine translation for it therefore lacks both training material and evaluation sets, and most existing dialect benchmarks link Arabic only to English. The authors started from OpenSubtitles-v2018 lines that were already aligned across MSA, English, French, German, Greek and Spanish. They filtered these lines aggressively, then had native speakers at a professional agency translate the remaining 120,600 lines into apc, working mostly from the MSA side. Because the other six languages were already aligned, one translation pass produced apc pairings with all of them. To test the resource, they trained a multilingual Marian model from scratch and fine-tuned Opus-MT and NLLB models, evaluating on FLORES-200 and on the Damascus/Aleppo portions of MADAR. Fine-tuning brought large gains on the short, conversational MADAR sentences. On the long, Wikipedia-style FLORES-200 sentences, gains were small or inconsistent. Human raters still ranked the fine-tuned dialect output highest. The study shows that dialect and domain are distinct factors that must each be handled.

Key points

  • The starting pool held 3,661,627 OpenSubtitles lines present in all six languages. Filters removed vulgarity, non-standard characters, lowercase-initial lines, near-duplicates, very short lines and frequency outliers, leaving 120,771 lines. A final cross-script check reduced this to 120,600 lines, released through LINDAT (hdl 11234/1-5033).
  • The corpus contains about 1M words on the English side and 738,812 words on the apc side. No orthographic standard was imposed; translators were asked only to be internally consistent.
  • A 1,000-sentence pilot translated both from English and from MSA showed no quality difference. The Overlap Coefficient between source MSA and output apc was 35.95 when translating from MSA and 26.85 when translating from English. This suggests the MSA source slightly reduced dialectal variety.
  • The systems compared were:
    • facebook/nllb-200-distilled-600M;
    • Helsinki-NLP Opus-MT models (76M–240M parameters);
    • a Marian transformer-big model (200M parameters) trained from scratch on all 84 directions with source tags.
  • Results were measured with chrF++ and COMET:
    • On MADAR, chrF++ into apc rose from 26.32 to 38.50 for Opus and from 33.04 to 37.77 for NLLB.
    • In the human ranking, raw inter-annotator agreement was 0.52 and Cohen's κ was 0.39.

Interesting ideas

  • I1. Picking only lines that are already aligned across many languages means a single human translation into the new variety yields many parallel pairs at once.
  • I2. A small pilot translated the same material from two different source languages, and lexical overlap was measured to see how much the source leaks into the output.
  • I3. A baseline that outputs the standard variety (MSA) was scored against dialect references, showing how far the nearest well-resourced variety gets without dialect-specific work.
  • I4. In the human ranking, the standard-variety output was hidden among the candidates, outputs were shuffled, and a shared control batch was used to compute agreement.
  • I5. Results were split into an in-domain test set and an out-of-domain one, which showed that dialect adaptation and domain shift act on different axes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Translation through the meta-world could be tested on such pivot-aligned sets, checking that the graphs built from each language version of one line match.
  • I2 (learn): Silver data produced by an LLM teacher could be checked the same way, measuring how strongly the prompt or source wording leaks into the generated text.
  • I3 (learn): For dialectal or informal English, a "parse it as standard English" baseline could serve as a negative control before adding dialect conversion rules.
  • I4 (learn): Human checks of Mova's explanations or answers could mix in an unlabelled plain-baseline output and a shared overlap subset, giving a measured κ instead of impressions.
  • I5 (answer): School-test and fable QA scores should be reported per domain, so that gains from tuning on tales are not mistaken for general reading ability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Charles UniversityPrague · Czechia133 works · 20,686 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mateusz Krubiński, Hashem Sellat, Shadi Saleh, Adam Pospíšil, Petr Zemánek and Pavel Pecina. 2023. Multi-Parallel Corpus of North Levantine Arabic. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.34

@inproceedings{krubinski2023multi,
  title = {{Multi-Parallel Corpus of North Levantine Arabic}},
  author = {Mateusz Krubiński, Hashem Sellat, Shadi Saleh, Adam Pospíšil, Petr Zemánek and Pavel Pecina},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.34},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.34}
}

References

25 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022. The Arabic parallel gender corpus 2.0: Extensions and analyses. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1870–1884, Marseille, France. European Language Resources Association.
  2. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131–198, Berlin, Germany. Association for Computational Linguistics. ↗
  3. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1240–1245, Reykjavik, Iceland. European Language Resources Association (ELRA).
  4. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA). find
  5. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. CoRR, abs/2207.04672. ↗
  6. Elham Abdullah Ghobain. 2017. Dubbing melodramas in the arab world; between the standard language and colloquial dialects. The Arabic Language and Literature, 2:49. ↗
  7. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al-Shargi, Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. 2018. Unified guidelines and resources for Arabic dialect orthography. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  8. Nizar Habash, Nasser Zalmout, Dima Taji, Hieu Hoang, and Maverick Alzate. 2017. A parallel corpus for evaluating machine translation between Arabic and European languages. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 235–241, Valencia, Spain. Association for Computational Linguistics.
  9. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Trans. Assoc. Comput. Linguistics, 5:339–351. ↗
  10. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  11. Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnik. 2018. A lexical distance study of arabic dialects. Procedia Computer Science, 142:2–13. Arabic Computational Linguistics. find
  12. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. ↗
  13. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  14. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. find
  15. El Moatez Billah Nagoudi, Ahmed El-Shangiti, Abdel-Rahim Elmadany, and Muhammad Abdul-Mageed. 2023. Dolphin: A challenging and diverse benchmark for arabic nlg.
  16. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  17. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  18. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗
  19. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
  20. Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020. AraBench: Benchmarking dialectal Arabic-English machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094–5107, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  21. Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for communication research. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 12, Number 3, September 2007: Special Issue on Invited Papers from ISCSLP 2006, pages 303–324. find
  22. Jörg Tiedemann. 2020. The tatoeba translation challenge – realistic data sets for low resource and multilingual MT. In Proceedings of the Fifth Conference on Machine Translation, pages 1174–1182, Online. Association for Computational Linguistics.
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
  24. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  25. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 49–59, Montréal, Canada. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/krubinski-2023-multi-parallel-corpus-north-levantine/