Elamur Research⌕
☰
home / papers / 2023 / Domain Adaptation for Arabic Machine Translation: The Case of Financial Texts

Domain Adaptation for Arabic Machine Translation: The Case of Financial Texts

Summary

The paper looks at how well machine translation from Arabic to English holds up in finance, a specialised domain that Arabic MT research had not yet covered. Earlier Arabic domain-adaptation work dealt mainly with news and medical text. The authors built what they describe as the first AR-EN parallel corpus of financial news, carefully aligned at the segment level, and used it as a benchmark. On this corpus they fine-tuned two pre-trained NMT models, Helsinki-NLP opus-mt-ar-en and NLLB-200-distilled-600M, and also fine-tuned ChatGPT-3.5 Turbo. They compared training on authentic pairs alone with training on data enlarged by machine-translated and synthetic pairs. Evaluation combined automatic metrics (lexical ones plus COMET and BERTScore) with ratings from human experts. Three findings stand out:

  • A small set of clean in-domain pairs is enough for fine-tuning to work.
  • ChatGPT-3.5 gave the best translations overall.
  • Adding augmented data generally made quality worse.

The work matters because it supplies public data and fine-tuned models for a domain and language pair with few resources, and because it is presented as the first study of adapting ChatGPT to the financial domain.

Key points

  • The resources are a new financial-news AR-EN parallel corpus and several fine-tuned checkpoints, published under the asas-ai organisation on Hugging Face. Examples include opus-mt-ar-en-finetuned-ar-to-en and nllb-200-distilled-600M variants trained on authentic, MT-augmented and synthetic-augmented data.
  • The fine-tuned models are OPUS-MT (ar-en), NLLB-200-distilled-600M and ChatGPT-3.5 Turbo. They were compared across three training setups.
  • When ChatGPT-3.5 was trained on augmented data, its COMET score fell to 23.03 while its BERTScore stayed high at 0.91. The authors read this as a disagreement between the two metrics.
  • Human evaluation worked as follows:
    • Three native-speaker finance postgraduates rated 50 random test sentences, seeing both source and translation.
    • They used a 1–4 acceptability scale running from Unacceptable to Ideal.
    • ChatGPT-3.5 received the top mean rating of 3.1 in setups #1 and #2.
    • In setup #3, the models were rated about equally.
  • BERTScore matched the human judgments more closely than COMET did. Of the NMT models, OPUS coped better with back-translated augmentation than with synthetic augmentation.

Interesting ideas

  • I1. A small, well-aligned in-domain set was enough for successful adaptation, while adding synthetic or machine-translated pairs tended to hurt, because their errors carry over into the model's outputs. — For Mova (learn): Prefer a small set of silver examples on which all witness parsers agree over a large set where they disagree, and test whether adding the lower-agreement data actually lowers parser accuracy.
  • I2. Augmentation sources were kept apart (back-translated versus synthetic), and a separate model was trained and released for each, so the effect of each source could be measured on its own. — For Mova (learn): Label each silver or LLM-teacher batch by where it came from and train ablations per source, so a harmful teacher tier can be identified and removed.
  • I3. When two automatic metrics disagreed (COMET versus BERTScore), the authors used human ratings to decide which metric to trust. — For Mova (back): Before choosing a single score for the text → graph → text compressor's output, check each automatic score against a small human-rated sample.
  • I4. Human raters saw the source alongside the output, scored on a four-level acceptability scale, and had to explain any extreme rating. This yields judgments that come with reasons. — For Mova (back): Use the same 1–4 rubric, with required justifications for extreme scores, when evaluating generated text and translations through the meta-world. The written reasons can then be mined for repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.12863 · arXiv 2309.12863

@misc{alghamdi2023domain,
  title = {{Domain Adaptation for Arabic Machine Translation: The Case of Financial Texts}},
  author = {Alghamdi, Emad A. and Zakraoui, Jezia and Abanmy, Fares A.},
  year = {2023},
  doi = {10.48550/arxiv.2309.12863},
  eprint = {2309.12863},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.12863}
}

References

65 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Philipp Koehn and Rebecca Knowles. “Six challenges for neural machine translation”. In: arXiv preprint arXiv:1706.03872 (2017). ↗
  2. Hal Daumé Iii and Jagadeesh Jagarlamudi. “Domain adaptation for machine translation by mining unseen words”. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011, pp. 407–412. find
  3. Danielle Saunders. “Domain adaptation and multi-domain adaptation for neural machine translation: A survey”. In: Journal of Artificial Intelligence Research 75 (2022), pp. 351–424. find
  4. Chenhui Chu and Rui Wang. “A survey of domain adaptation for machine translation”. In: Journal of information processing 28 (2020), pp. 413–426. find
  5. Yasmin Moslem, Rejwanul Haque, and Andy Way. “Adaptive machine translation with large language models”. In: arXiv preprint arXiv:2301.13294 (2023). ↗ our card
  6. Martin Popel et al. “Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals”. In: Nature communications 11.1 (2020), pp. 1–15. find
  7. Arwa Hatem and Nazlia Omar. “Syntactic reordering for Arabic-English phrase-based machine translation”. In: Database theory and application, bio-science and bio-technology. Springer, 2010, pp. 198–206. find
  8. Zakaryia Mustafa Almahasees. “Assessment of Google and Microsoft Bing translation of journalistic texts”. In: International Journal of Languages, Literature and Linguistics 4.3 (2018), pp. 231–235. find
  9. Rico Sennrich, Barry Haddow, and Alexandra Birch. “Neural machine translation of rare words with subword units”. In: arXiv preprint arXiv:1508.07909 (2015). ↗
  10. Marta R Costa-Jussa and José AR Fonollosa. “Character-based neural machine translation”. In: arXiv preprint arXiv:1603.00810 (2016). ↗ our card
  11. Minh-Thang Luong and Christopher D Manning. “Achieving open vocabulary neural machine translation with hybrid word-character models”. In: arXiv preprint arXiv:1604.00788 (2016). ↗ our card
  12. Mathias Müller, Annette Rios, and Rico Sennrich. “Domain robustness in neural machine translation”. In: arXiv preprint arXiv:1911.03109 (2019). ↗ our card
  13. Mai Oudah, Amjad Almahairi, and Nizar Habash. “The impact of preprocessing on Arabic-English statistical and neural machine translation”. In: arXiv preprint arXiv:1906.11751 (2019). ↗
  14. Yasmin Moslem et al. “Domain-Specific Text Generation for Machine Translation”. In: arXiv preprint arXiv:2208.05909 (2022). ↗
  15. Kishore Papineni et al. “BLEU: a method for automatic evaluation of machine translation”. In: Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics. 2002, pp. 311–318. find our card
  16. Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. “Simple, scalable adaptation for neural machine translation”. In: arXiv preprint arXiv:1909.08478 (2019). ↗ our card
  17. Shuoheng Yang, Yuxin Wang, and Xiaowen Chu. “A survey of deep learning techniques for neural machine translation”. In: arXiv preprint arXiv:2002.07526 (2020). ↗
  18. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. “Sequence to sequence learning with neural networks”. In: Advances in neural information processing systems 27 (2014). find
  19. Melvin Johnson et al. “Google’s multilingual neural machine translation system: Enabling zero-shot translation”. In: Transactions of the Association for Computational Linguistics 5 (2017), pp. 339–351. find
  20. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. “Neural machine translation by jointly learning to align and translate”. In: arXiv preprint arXiv:1409.0473 (2014). ↗ our card
  21. Jie Zhou et al. “Deep recurrent models with fast-forward connections for neural machine translation”. In: Transactions of the Association for Computational Linguistics 4 (2016), pp. 371–383. find
  22. Duygu Ataman, Wilker Aziz, and Alexandra Birch. “A latent morphology model for open-vocabulary neural machine translation”. In: arXiv preprint arXiv:1910.13890 (2019). ↗
  23. Fandong Meng et al. “Encoding source language with convolutional neural network for machine translation”. In: arXiv preprint arXiv:1503.01838 (2015). ↗
  24. Jonas Gehring et al. “A convolutional encoder model for neural machine translation”. In: arXiv preprint arXiv:1611.02344 (2016). ↗ our card
  25. Nal Kalchbrenner et al. “Neural machine translation in linear time”. In: arXiv preprint arXiv:1610.10099 (2016). ↗
  26. Jonas Gehring et al. “Convolutional sequence to sequence learning”. In: International conference on machine learning. PMLR. 2017, pp. 1243–1252. find our card
  27. Ashish Vaswani et al. “Attention is all you need”. In: Advances in neural information processing systems 30 (2017). find
  28. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. “Effective approaches to attention-based neural machine translation”. In: arXiv preprint arXiv:1508.04025 (2015). ↗
  29. Felix Stahlberg. “Neural machine translation: A review”. In: Journal of Artificial Intelligence Research 69 (2020), pp. 343–418. find
  30. Jacob Devlin et al. “Bert: Pre-training of deep bidirectional transformers for language understanding”. In: arXiv preprint arXiv:1810.04805 (2018). ↗
  31. Wissam Antoun, Fady Baly, and Hazem Hajj. “Arabert: Transformer-based model for Arabic language understanding”. In: arXiv preprint arXiv:2003.00104 (2020). ↗ our card
  32. Wuwei Lan et al. “Gigabert: Zero-shot transfer learning from english to arabic”. In: Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP). 2020. find
  33. Sadaf Abdul-Rauf et al. “Exploring transfer learning and domain data selection for the biomedical translation”. In: Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2). 2019, pp. 156–163. find
  34. Boxiang Liu and Liang Huang. “ParaMed: a parallel corpus for English–Chinese translation in the biomedical domain”. In: BMC Medical Informatics and Decision Making 21.1 (2021), p. 258. find
  35. Rubén Martínez-Domínguez et al. “Customized Neural Machine Translation Systems for the Swiss Legal Domain”. In: Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 2: User Track). 2020, pp. 217–223. find
  36. Samuel Läubli et al. “Post-editing productivity with neural machine translation: An empirical assessment of speed and quality in the banking and finance domain”. In: arXiv preprint arXiv:1906.01685 (2019). ↗
  37. David Grangier and Dan Iter. “The trade-offs of domain adaptation for neural language models”. In: arXiv preprint arXiv:2109.10274 (2021). ↗
  38. Alberto Poncelas et al. Investigating Backtranslation in Neural Machine Translation. 2018. 1804. find our card
  39. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. “An empirical comparison of domain adaptation methods for neural machine translation”. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2017, pp. 385–391. find our card
  40. Yongchao Deng et al. “Factorized transformer for multi-domain neural machine translation”. In: Findings of the Association for Computational Linguistics: EMNLP 2020. 2020, pp. 4221–4230. find
  41. Hitham Abo Bakr, Khaled Shaalan, and Ibrahim Ziedan. “A hybrid approach for converting written Egyptian colloquial dialect into diacritized Arabic”. In: The 6th international conference on informatics and systems, infos2008. cairo university. Citeseer. 2008. find
  42. Emad Mohamed, Behrang Mohit, and Kemal Oflazer. “Transforming standard Arabic to colloquial Arabic”. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2012, pp. 176–180. find
  43. Nizar Habash and Jun Hu. “Improving Arabic-Chinese statistical machine translation using English as pivot language”. In: Proceedings of the Fourth Workshop on Statistical Machine Translation. 2009, pp. 173–181. find
  44. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. “AraT5: Text-to-text transformers for Arabic language generation”. In: arXiv preprint arXiv:2109.12068 (2021). ↗ our card
  45. Jezia Zakraoui et al. “Arabic Machine Translation: A Survey With Challenges and Future Directions”. In: IEEE Access 9 (2021), pp. 161445–161468. find
  46. Mohamed Seghir Hadj Ameur, Farid Meziane, and Ahmed Guessoum. “Arabic machine translation: A survey of the latest trends and challenges”. In: Computer Science Review 38 (2020), p. 100305. find
  47. Nizar Y Habash. “Introduction to Arabic natural language processing”. In: Synthesis lectures on human language technologies 3.1 (2010), pp. 1–187. find
  48. Rana Ehab, Eslam Amer, and Mahmoud Gadallah. “English-Arabic hybrid machine translation system using EBMT and translation memory”. In: Int. J. Adv. Comput. Sci. Appl. 10.1 (2019), pp. 195–203. find
  49. Wissam Antoun, Fady Baly, and Hazem Hajj. “AraGPT2: Pre-Trained Transformer for Arabic Language Generation”. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop. Kyiv, Ukraine (Virtual): Association for Computational Linguistics, Apr. 2021, pp. 196–207. URL : https://aclanthology.org/2021.wanlp-1.21. ↗
  50. Alec Radford et al. “Language Models are Unsupervised Multitask Learners”. In: (2019). find
  51. Md Tawkat Islam Khondaker et al. GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP. 2023. arXiv: 2305.14976 [cs.CL]. ↗
  52. Jörg Tiedemann and Santhosh Thottingal. “OPUS-MT – Building open translation services for the World”. In: European Association for Machine Translation Conferences/Workshops. 2020. URL : https://api.semanticscholar.org/CorpusID:221097277. ↗
  53. Jörg Tiedemann. “The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT”. In: Proceedings of the Fifth Conference on Machine Translation. Online: Association for Computational Linguistics, Nov. 2020, pp. 1174–1182. URL : https://www.aclweb.org/anthology/2020.wmt-1.139. ↗ our card
  54. Taku Kudo and John Richardson. “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing”. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 - November 4, 2018. 2018, pp. 66–71. DOI : 10.18653/v1/d18-2012. URL : https://doi.org/10.18653/v1/d18-2012. ↗ our card
  55. NLLB Team et al. No Language Left Behind: Scaling Human-Centered Machine Translation. 2022. arXiv: 2207.04672 [cs.CL]. ↗
  56. Maja Popović. “chrF: character n-gram F-score for automatic MT evaluation”. In: Proceedings of the Tenth Workshop on Statistical Machine Translation. Lisbon, Portugal: Association for Computational Linguistics, Sept. 2015, pp. 392–395. DOI : 10.18653/v1/W15-3049. URL : https://aclanthology.org/W15-3049. ↗ our card
  57. Matthew Snover et al. “A study of translation edit rate with targeted human annotation”. In: Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers. 2006, pp. 223–231. find our card
  58. Ricardo Rei et al. “COMET: A Neural Framework for MT Evaluation”. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, Nov. 2020, pp. 2685–2702. DOI : 10 . 18653 / v1 / 2020 . emnlp - main . 213. URL : https://aclanthology.org/2020.emnlp-main.213. ↗ our card
  59. Tianyi Zhang et al. BERTScore: Evaluating Text Generation with BERT. 2020. arXiv: 1904.09675 [cs.CL]. ↗
  60. Keqin Peng et al. “Towards Making the Most of ChatGPT for Machine Translation”. In: ArXiv abs/2303.13780 (2023). URL : https://api.semanticscholar.org/CorpusID:257704711. ↗ our card
  61. Faten Khoshafah. ChatGPT for Arabic-English Translation: Evaluating the Accuracy. 2023. find
  62. Zaid Alyafeai et al. “Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models”. In: arXiv preprint arXiv:2306.16322 (2023). ↗ our card
  63. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. AraT5: Text-to-Text Transformers for Arabic Language Generation. 2022. arXiv: 2109.12068 [cs.CL]. ↗ our card
  64. Amr Hendy et al. How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. 2023. arXiv: 2302.09210 [cs.CL]. ↗
  65. Deborah Coughlin. “Correlating automated and human assessments of machine translation quality”. In: Proceedings of Machine Translation Summit IX: Papers. 2003. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alghamdi-2023-domain-adaptation-arabic-machine-translation/