Elamur Research⌕
☰
home / papers / 2023 / Dolphin: A Challenging and Diverse Benchmark for Arabic NLG

Dolphin: A Challenging and Diverse Benchmark for Arabic NLG

Paper · 2023 · El Moatez Billah Nagoudi, AbdelRahim Elmadany, Ahmed Oumar El-Shangiti et al.

Summary

Arabic is a large family of varieties, from Classical Arabic through Modern Standard Arabic (MSA) to many regional dialects, and it is also written in Latin script (Arabizi). Before this paper, it had no sizeable shared benchmark for natural language generation. The authors built Dolphin to fill that gap. It gathers 40 publicly available datasets, a few of them newly created by the team, into 13 task clusters with 50 test splits. Tasks include machine translation, summarization, question answering and generation, dialogue, paraphrase, diacritization, transliteration, grammatical error correction, code-switching and data-to-text.

Each task is scored with a metric suited to it, and the results are combined into a single Dolphin score. That score is shown on a public, modular leaderboard that also stores detailed metadata about the data and the models.

Arabic-dedicated and multilingual models were evaluated as baselines. A finetuned AraT5 v2 usually beat prompted ChatGPT and BLOOMZ. The benchmark gives Arabic NLG a common yardstick and shows which varieties and tasks still lack resources.

Key points

  • Scale and comparison:
    • Dolphin has 40 datasets, 13 task clusters and 50 test splits.
    • The authors say this makes it the largest Arabic NLG benchmark.
    • It is also larger than earlier ones such as ARGEN (19 datasets, 7 tasks) and AraBench.
  • Varieties and scripts: coverage includes Arabizi, CA, DA and MSA.
    • Tasks are grouped by direction, for example DA → MSA, Arabizi → X, Table → MSA and X → MSA.
    • The X → MSA translation task uses UN corpus dev/test sets, with 50K training pairs per language sampled from MultiUN.
  • Metrics:
    • BLEU, ROUGE and F1 are used for most tasks.
    • MaxMatch (M2) is used for grammatical error correction.
    • Character Error Rate (CER) is used for diacritization.
  • Model results:
    • Finetuned AraT5 v2 beat ChatGPT and BLOOMZ in 14 of 16 comparisons.
    • ChatGPT with 10 examples won both code-switching tasks, plausibly because of its multilingual pretraining.
    • BLOOMZ contains only about 1% Arabic.
  • Stated limitations:
    • There is no data for the dialects of Djibouti, Mauritania or Yemen.
    • Two datasets (AEC and AraPara) were produced by machine translation and may carry bias.
    • Automatic metrics fall short for open-ended dialogue, where human evaluation is needed.

Interesting ideas

  • I1. Label every task by its source and target variety (e.g., dialect → standard) and report scores per direction, so weaknesses tied to one register stay visible instead of being averaged away.
  • I2. Treat nonstandard spellings such as Arabizi as a separate input condition with its own test sets rather than as noise, which forces systems to handle how people really type.
  • I3. Match the metric to the task, using CER for character-level restoration and M2 for error correction, instead of forcing BLEU everywhere.
  • I4. Make the leaderboard store rich metadata (split sizes, data sources, parameter counts, epochs to convergence, finetuning details) so results can be reproduced and placed in context.
  • I5. Compare a small finetuned model against a few-shot large LLM task by task; the LLM wins only on specific phenomena such as code-switching.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Split parser evaluation by register (formal prose, chat, transcribed speech, older literary English) to see where the UD trees degrade.
  • I2 (tokens): Add a test slice of informal, phonetically spelled English so that tokenizer and lemma errors on such input are measured separately.
  • I3 (back): Score the text → graph → text compressor with edit-based and meaning-preservation measures suited to each output type, not with n-gram overlap alone.
  • I4 (learn): Attach a small metadata record to every evaluation run (silver data source, number of witness parsers, training epochs) so that comparisons between runs stay honest.
  • I5 (learn): Use per-task comparisons like this to decide on which phenomena the LLM teacher tier is worth calling and where Mova's own trained components are already better.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksText generation · 202 works

Cite

Cite as El Moatez Billah Nagoudi, AbdelRahim Elmadany, Ahmed Oumar El-Shangiti and Muhammad Abdul-Mageed. 2023. Dolphin: A Challenging and Diverse Benchmark for Arabic NLG. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.98

@inproceedings{nagoudi2023dolphin,
  title = {{Dolphin: A Challenging and Diverse Benchmark for Arabic NLG}},
  author = {El Moatez Billah Nagoudi, AbdelRahim Elmadany, Ahmed Oumar El-Shangiti and Muhammad Abdul-Mageed},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.98},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.98}
}

References

68 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022. User-Centric Gender Rewriting. In Proceedings of the 2022 Conference of the North American https://alliancecan.ca https://arc.ubc.ca/ubc-arc-sockeye 16 https://sites.research.google/trc/about/Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 618– 631, Seattle, United States. Association for Computational Linguistics. ↗
  2. Marwah Alian, Arafat Awajan, Ahmad Al-Hasan, and Raeda Akuzhia. 2019. Towards building arabic paraphrasing benchmark. In Proceedings of the Second International conference on Data Science E-learning and Information Systems (DATA’ 2019), pages 1–5. find
  3. Mohamed Seghir Hadj Ameur, Farid Meziane, and Ahmed Guessoum. 2019. Anetac: Arabic named entity transliteration and classification dataset. arXiv preprint arXiv:1907.03110. ↗
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. ↗
  5. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637.
  6. Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, Yuan-Fang Li, Yong bin Kang, and Rifat Shahriyar. 2021. Crosssum: Beyond english-centric cross-lingual abstractive text summarization for 1500+ language pairs. CoRR, abs/2112.08804. ↗
  7. Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, and Rifat Shahriyar. 2023. BanglaNLG and BanglaT5: Benchmarks and resources for evaluating low-resource natural language generation in Bangla. pages 726–735. find
  8. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A Multidialectal Parallel Corpus of Arabic. In LREC, pages 1240–1245.
  9. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The Madar Arabic Dialect Corpus and Lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). find
  10. Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Leylia Khodra, et al. 2021. Indonlg: Benchmark and resources for evaluating indonesian natural language generation. pages 8875–8898. find
  11. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055. ↗
  12. Yiran Chen, Zhenqiao Song, Xianze Wu, Danqing Wang, Jingjing Xu, Jiaze Chen, Hao Zhou, and Lei Li. 2022. MTG: A benchmark suite for multilingual text generation. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2508–2527, Seattle, United States. Association for Computational Linguistics. ↗
  13. Amina Chouigui, Oussama Ben Khiroun, and Bilel Elayeb. 2021. An arabic multi-source news corpus: Experimenting on single-document extractive summarization. Arabian Journal for Science and Engineering, 46:3925–3938. ↗
  14. Aleksandr Chuklin, Justin Zhao, and Mihir Kale. 2022. Clse: Corpus of linguistically significant entities. pages 78–96. find
  15. Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568–572. find
  16. Kareem Darwish. 2013. Arabizi detection and conversion to arabic. arXiv preprint arXiv:1306.6755. ↗
  17. Long Doan, Linh The Nguyen, Nguyen Luong Tran, Thai Hoang, and Dat Quoc Nguyen. 2021. PhoMT: A high-quality and large-scale benchmark dataset for Vietnamese-English machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4495–4503, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  18. Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022. Arabart: a pretrained arabic sequence-to-sequence model for abstractive summarization.
  19. Andreas Eisele and Yu Chen. 2010. MultiUN: A multilingual corpus from united nation documents. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA).
  20. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. ORCA: A Challenging Benchmark for Arabic Language Understanding.
  21. Mohamed Elmahdy, Mark Hasegawa-Johnson, and Eiman Mustafawi. 2014. Development of a TV broadcasts speech recognition system for qatari Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3057–3061, Reykjavik, Iceland. European Language Resources Association (ELRA).
  22. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019. Arabic text diacritization using deep neural networks.
  23. Kamel Gaanoun, Abdou Naira, Anass Allak, and Imade Benelallam. 2022. Automatic Text Summarization for Moroccan Arabic Dialect Using an Artificial Intelligence Approach, pages 158–177. find
  24. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. pages 96–120. find
  25. Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, and Bernd Bohnet. 2022. GEMv2: Multilingual NLG benchmarking in a single line of code. pages 266–281. find
  26. Jian Guan, Zhuoer Feng, Yamei Chen, Ruilin He, Xiaoxi Mao, Changjie Fan, and Minlie Huang. 2022. LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation. Transactions of the Association for Computational Linguistics, 10:434–451. ↗
  27. Tri Wahyu Guntara, Alham Fikri Aji, and Radityo Eko Prasojo. 2020. Benchmarking multidomain English-Indonesian machine translation. In Proceedings of the 13th Workshop on Building and Using Comparable Corpora, pages 35–43, Marseille, France. European Language Resources Association. find
  28. Nizar Habash and David Palfreyman. 2022. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88, Marseille, France. European Language Resources Association. find
  29. Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5427–5444, Online. Association for Computational Linguistics. find
  30. Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. Xl-sum: Large-scale multilingual abstractive summarization for 44 languages. ↗
  31. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411–4421. PMLR. find
  32. Walaa Ismail and Masun Nabhan Homsi. 2018. Dawqas: A dataset for arabic why question answering system. Procedia Computer Science, 142:123–131. find
  33. Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra, and Pratyush Kumar. 2022. IndicNLG benchmark: Multilingual datasets for diverse NLG tasks in Indic languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5363–5394, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  34. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. pages 7871–7880.
  35. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. Mlqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475. ↗
  36. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995, Taipei, Taiwan. Asian Federation of Natural Language Processing. ↗
  37. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out.
  38. Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, and Nan Duan. 2021. GLGE: A new general language generation evaluation benchmark. pages 408–420. find
  39. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. ↗
  40. Yuval Merhav and Stephen Ash. 2018. Design Challenges in Named Entity Transliteration. In Proceedings of the 27th International Conference on Computational Linguistics, pages 630–640, Santa Fe, New Mexico, USA. Association for Computational Linguistics. ↗
  41. Simon Mille, Anya Belz, Bernd Bohnet, Thiago Castro Ferreira, Yvette Graham, and Leo Wanner. 2020. The third multilingual surface realisation shared task (SR’20): Overview and evaluation results. In Proceedings of the Third Workshop on Multilingual Surface Realisation, pages 1–20, Barcelona, Spain (Online). Association for Computational Linguistics.
  42. Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. 2014. The first QALB shared task on automatic text correction for Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 39–47, Doha, Qatar. Association for Computational Linguistics.
  43. Andrew Morris, Viktoria Maier, and Phil Green. 2004. From wer and ril to mer and wil: improved evaluation measures for connected speech recognition. find
  44. Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj. 2019. Neural arabic question answering. arXiv preprint arXiv:1906.05394. ↗
  45. Hamdy Mubarak. 2018. Dial2msa: A tweets corpus for converting dialectal arabic to modern standard arabic. In OSACT 3: The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, page 49. find
  46. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual generalization through multitask finetuning. ↗
  47. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022a. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  48. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022b. TURJUMAN: A public toolkit for neural Arabic machine translation. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 1–11, Marseille, France. European Language Resources Association. ↗
  49. Tarek Naous, Wissam Antoun, Reem Mahmoud, and Hazem Hajj. 2021. Empathetic BERT2BERT conversational model: Learning Arabic language generation with little data. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 164– 172, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  50. Tarek Naous, Zahraa Bassyouni, Bassel Mousi, Hazem Hajj, Wassim El Hajj, and Khaled Shaban. 2023. Open-domain response generation in low-resource settings using self-supervised pre-training of warm-started transformers. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4):1–12. ↗
  51. Tarek Naous, Christian Hokayem, and Hazem Hajj. 2020. Empathy-driven arabic conversational chatbot. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 58–68.
  52. Aissam Outchakoucht and Hamza Es-Samaali. 2021. Moroccan dialect -darija- open dataset. ↗
  53. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. ↗
  54. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. ↗
  55. Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020. LAReQA: Language-agnostic answer retrieval from a multilingual pool. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5919–5930, Online. Association for Computational Linguistics. find
  56. Alla Rozovskaya, Houda Bouamor, Nizar Habash, Wajdi Zaghouani, Ossama Obeid, and Behrang Mohit. 2015. The second QALB shared task on automatic text correction for Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 26–35, Beijing, China. Association for Computational Linguistics.
  57. Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020. AraBench: Benchmarking dialectal Arabic-English machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094–5107, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  58. Yves Scherrer. 2020. TaPaCo: A corpus of sentential paraphrases for 73 languages. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6868–6873, Marseille, France. European Language Resources Association. find
  59. Djamé Seddah, Farah Essaidi, Amal Fethi, Matthieu Futeral, Benjamin Muller, Pedro Javier Ortiz Suárez, Benoît Sagot, and Abhishek Srivastava. 2020. Building a user-generated content North-African Arabizi treebank: Tackling hell. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1139–1150, Online. Association for Computational Linguistics. find
  60. Zhiyi Song, Stephanie M Strassel, Haejoong Lee, Kevin Walker, Jonathan Wright, Jennifer Garland, Dana Fore, Brian Gainor, Preston Cabe, Thomas Thomas, et al. 2014. Collecting natural sms and chat conversations in multiple languages: The bolt phase 2 corpus. In LREC, pages 1699–1704. Citeseer. find
  61. Bashar Talafha, Analle Abuammar, and Mahmoud Al-Ayyoub. 2021. Atar: Attention-based lstm for arabizi transliteration. International Journal of Electrical and Computer Engineering, 11:2327–2334. ↗
  62. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. pages 2214–2218. find
  63. Daniel Varab and Natalie Schluter. 2021. MassiveSumm: a very large-scale, very multilingual, news summarisation dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10150–10161, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  64. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. ↗
  65. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. ↗
  66. Yuan Yao, Qingxiu Dong, Jian Guan, Boxi Cao, Zhengyan Zhang, Chaojun Xiao, Xiaozhi Wang, Fanchao Qi, Junwei Bao, Jinran Nie, et al. 2021. CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark. arXiv e-prints, pages arXiv– 2112. find
  67. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 49–59.
  68. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1. 0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nagoudi-2023-dolphin-challenging-diverse-benchmark-arabic/