Elamur Research⌕
☰
home / papers / 2023 / FinAraT5: A text to text model for financial Arabic text understanding and generation

FinAraT5: A text to text model for financial Arabic text understanding and generation

Paper · 2023 · Nadhem Zmandar, Mo El-Haj and Paul Rayson

Summary

Arabic has few pretrained language models, and almost none target finance. Most existing Arabic models are encoder-only and general-purpose. The authors build FinAraT5 by continuing the pre-training of the existing araT5 checkpoint on a financial Arabic corpus they assembled themselves. The corpus combines about 30,000 PDF disclosures from Gulf stock exchanges with roughly 30,000 economic news items. Because FinAraT5 is an encoder-decoder (text-to-text) model, it handles both classification and generation. They test it on two tasks. The first is a financial sentiment task translated from English. The second is FinAraSum, a new summarisation dataset they deliberately made highly abstractive, since earlier Arabic summarisation benchmarks mostly reward copying. FinAraT5 scores best on sentiment and is competitive in summarisation against mT5, AraBART, BERT-based models and the original araT5. Its summaries repeat less than those of most baselines. The work shows that cheap domain adaptation of an existing seq2seq model can serve an under-resourced language and domain. The models are released on Hugging Face.

Key points

  • FinAraT5 has about 220M parameters. It was trained for 45 days on a Google Cloud TPU v3.8, starting from araT5 base.
  • The source data is around 30,000 PDFs (roughly 25 GB) from TASI, NOMU, DFM, Abu Dhabi, Boursa Kuwait, Muscat, Qatar and Bahrain exchanges, covering more than 35 sectors.
  • The PDFs were hard to convert to text because of scans, old Arabic fonts and heavy tables. OCR was slow and had a low success rate for Arabic.
  • On sentiment accuracy, FinAraT5 reached 0.9449, against 0.9362 for araT5 and 0.9246 for both arabert and mT5.
  • On FinAraSum, summaries average about 9 tokens.
    • Repetition (summaries containing any of the corpus's 400 most frequent words) is 3.48% for FinAraT5, 0.52% for gold and 15.76% for bert2bert.
    • Novel unigrams make up 28.8% of FinAraT5's output versus 37.1% of the gold summaries.

Interesting ideas

  • I1. Instead of training from scratch, the authors adapt a general checkpoint by continuing pre-training on in-domain text, which is far cheaper and keeps general language ability.
  • I2. They measure how abstractive a summary is by counting n-grams absent from the source, reported for both system and gold output. This gives a simple yardstick for copying versus rephrasing.
  • I3. They define a repetition metric as the share of outputs containing words from a fixed frequent-word list, a cheap detector of degenerate, generic generation.
  • I4. They built a new benchmark on purpose because existing datasets were extractive, so they could not distinguish real abstraction from copying.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): the parsers could first be trained on general treebanks and then retrained on silver data from a target genre (fables, school texts), instead of being rebuilt for each genre.
  • I2 (back): when the text → graph → text compressor regenerates text, novel-n-gram rates could show whether it truly verbalises the meaning graph or just echoes surface strings.
  • I3 (back): a similar frequent-word check on generated answers or summaries could flag bland, template-like output from the generation stage.
  • I4 (answer): QA test items could be chosen so the answer is not a literal span of the text, which forces the readers to use the world model rather than string matching.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Nadhem Zmandar, Mo El-Haj and Paul Rayson. 2023. FinAraT5: A text to text model for financial Arabic text understanding and generation. Proceedings of the 4th Conference on Language, Data and Knowledge.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zmandar2023finarat5,
  title = {{FinAraT5: A text to text model for financial Arabic text understanding and generation}},
  author = {Nadhem Zmandar, Mo El-Haj and Paul Rayson},
  booktitle = {Proceedings of the 4th Conference on Language, Data and Knowledge},
  year = {2023},
  url = {https://aclanthology.org/2023.ldk-1.25/}
}

References

25 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abbas Ghaddar, Yimeng Wu, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Duan Xinyu, Zhefeng Wang, Baoxing Huai, Xin Jiang, Qun Liu, and Philippe Langlais. 2021. JABER: junior arabic bert. CoRR, abs/2112.04329. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Zaid Alyafeai and Maged Saeed. 2020. tkseem: A preprocessing library for arabic. https://github.com/ARBML/tnkeeh. ↗
  4. Moussa Kamal Eddine, Guokan Shang, Antoine Tixier, and Michalis Vazirgiannis. 2022a. FrugalScore: Learning cheaper, lighter and faster evaluation metrics for automatic text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1305–1318, Dublin, Ireland. Association for Computational Linguistics. find
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  6. Moussa Kamal Eddine, Antoine Tixier, and Michalis Vazirgiannis. 2021. BARThez: a skilled pretrained French sequence-to-sequence model. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9369–9390, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  7. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022b. AraBART: a pretrained Arabic sequence-to-sequence model for abstractive summarization. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 31–42, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. 19 https://sites.research.google/trc/ about/ 20 https://cloud.google.com/edu/ researchers Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. ↗
  8. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  9. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. ↗
  10. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL. ↗
  11. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗
  12. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762. ↗
  13. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. ↗
  14. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  15. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. ↗
  16. Yi Yang, Mark Christopher Siy UY, and Allen Huang. 2020. Finbert: A pretrained language model for financial communications. ↗
  17. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation.
  18. Pekka Malo, Ankur Sinha, Pyry Takala, Pekka J. Korhonen, and Jyrki Wallenius. 2013. Good debt or bad debt: Detecting semantic orientations in economic texts. CoRR, abs/1307.5336. ↗
  19. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy. Association for Computational Linguistics. ↗
  20. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  21. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. arXiv:1904.09675 [cs]. ArXiv: 1904.09675. ↗
  22. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics. ↗
  23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019a. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683. ↗
  24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019b. Exploring the limits of transfer learning with a unified text-to-text transformer. ↗
  25. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264–280. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zmandar-2023-finarat5-text-text-model-financial/