Elamur Research⌕
☰
home / papers / 2020 / Arabizi Language Models for Sentiment Analysis

Arabizi Language Models for Sentiment Analysis

Paper · 2020 · Gaétan Baert, Souhir Gahbiche, Guillaume Gadek et al.

Summary

Arabizi is Arabic, either dialect or MSA, typed with Latin letters plus digits that stand for sounds the Latin alphabet lacks. It is the main form of informal Arabic on social networks. It has no spelling norms and is often mixed with English or French inside a single sentence, and mainstream pretrained models such as multilingual BERT are not built for it.

The authors fill the resource gap with two releases:

  • LAD, a corpus of 7.7M Arabizi tweets.
  • SALAD, a hand-labelled sentiment subset of LAD. Its labels include cases of mixed feelings and cases where a tweet's words disagree with its emojis.

On LAD they pretrain BAERT, a BERT-style model, in three variants. The variants separate the effect of inherited weights from the effect of the vocabulary. After fine-tuning for sentiment classification, the Arabizi models clearly beat multilingual BERT on SALAD and on the Tunisian TUNIZI set. The vocabulary fitted to Arabizi gives the largest gains on SALAD. Reusing multilingual weights mainly speeds up convergence. The work gives Arabizi research a shared, sizeable starting point.

Key points

  • Resources: LAD (7.7M tweets) and SALAD (manually annotated subset) are released. SALAD is mostly Egyptian Arabizi mixed with English. TUNIZI (about 3,000 comments) is Tunisian Arabizi mixed mostly with French.
  • Compared models: BMU (multilingual BERT) and three BAERT variants:
    • BAERT-bmu-retrain: BMU weights and BMU vocabulary, further pretrained on LAD.
    • BAERT-bmu-scratch: BMU vocabulary, trained from scratch.
    • BAERT-arz-scratch: new Arabizi vocabulary, trained from scratch.
  • F1 results:
    • SALAD, 5 classes: 41.6 (BMU) vs 59.9 (BAERT-arz-scratch).
    • SALAD, 3 classes: 58.9 vs 74.4.
    • TUNIZI: BAERT-bmu-retrain is best at 83.8, against 82.7 for arz-scratch. That gap is smaller than the standard deviation.
  • Tokenizer statistics on LAD: the Arz-vocabulary needs 1.67 tokens per word against 1.97 for BMU, and its [UNK] rate is 0.6% against 3.1%. On TUNIZI the picture reverses: BMU's [UNK] rate is 0.01%.
  • Training cost and weak classes: starting from BMU weights halved pretraining (200k vs 400k batches of 64). The Conflict class (about 5% of the data) stayed at or below 5% F1 for every model.

Interesting ideas

  • I1. Cheap tokenizer statistics (tokens per word, share of unknown tokens) are proposed as a signal for choosing between training from scratch and reusing pretrained weights, before paying for expensive training.
  • I2. The ablation design changes one factor at a time (weights vs vocabulary), so a gain can be attributed to its real cause rather than to the model as a whole.
  • I3. The annotation scheme gives a separate label to tweets whose text and emojis point in opposite directions, treating that clash as a phenomenon of its own.
  • I4. Results are reported at several label granularities (5, 4 and 3 classes) by merging hard classes, which shows where the difficulty actually sits.
  • I5. Mean and standard deviation over runs are given, and the authors refuse to call a winner when the gap is below the spread.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Track unknown-word and split rates of the en tokenizer and lemmatizer on each new text genre, as an early warning that the lexicon needs extending.
  • I2 (learn): When retraining the perceptron parser or gparse, run paired variants that differ only in the feature or lexicon set versus the initial weights, so improvements can be explained.
  • I3 (frames): The irony detector in prag could explicitly flag disagreement between surface polarity cues and the stated content, as an explainable trigger for an ironic reading.
  • I4 (answer): Score readers on fable QA at both fine and merged answer categories, to see whether errors come from rare distinctions or from basic understanding.
  • I5 (learn): Require multi-seed runs with standard deviation in Mova's evaluation reports, and treat sub-noise differences as ties before accepting a repair rule or a new model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Institut National des Sciences Appliquées Rouen Normandie · Airbus (France) · Laboratoire d'Informatique, du Traitement de l'Information et des Systèmes.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSentiment & emotion · 972 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Gaétan Baert, Souhir Gahbiche, Guillaume Gadek and Alexandre Pauchet. 2020. Arabizi Language Models for Sentiment Analysis. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.51

@inproceedings{baert2020arabizi,
  title = {{Arabizi Language Models for Sentiment Analysis}},
  author = {Gaétan Baert, Souhir Gahbiche, Guillaume Gadek and Alexandre Pauchet},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.51},
  url = {https://doi.org/10.18653/v1/2020.coling-main.51}
}

References

39 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abbasi, Hsinchun Chen, and Arab Salem. 2008. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM Transactions on Information Systems (TOIS), 26(3):12. find
  2. Wid H. Allehaiby. 2013. Arabizi: An analysis of the Romanization of the Arabic Script from a Sociolinguistic Perspective. Arab World English Journal (AWEJ), 4(3):52–62. find
  3. Ann Bies, Zhiyi Song, Mohamed Maamouri, Stephen Grimes, Haejoong Lee, Jonathan Wright, Stephanie Strassel, Nizar Habash, Ramy Eskander, and Owen Rambow. 2014. Transliteration of arabizi into arabic orthography: Developing a parallel annotated arabizi-arabic script sms/chat corpus. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 93–103.
  4. Cagatay Catal and Mehmet Nangir. 2017. A sentiment classification model based on multiple classifiers. Applied Soft Computing, 50:135–141. ↗
  5. J. Chung, C. Gulcehre, K.-H. Cho, and Y. Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. ↗
  6. Ryan Cotterell and Chris Callison-Burch. 2014. A multi-dialect, multi-genre corpus of informal written Arabic. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, may. European Language Resources Association (ELRA).
  7. Ryan Cotterell, Adithya Renduchintala, Naomi Saphra, and Chris Callison-Burch. 2014. An algerian arabic-french code-switched corpus. In LREC Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools. find
  8. Kareem Darwish. 2014. Arabizi Detection and Conversion to Arabic. ANLP 2014, page 217. ↗
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  10. R. M. Duwairi, M. Alfaqeh, M. Wardat, and A. Alrabadi. 2016. Sentiment analysis for Arabizi text. In 2016 7th International Conference on Information and Communication Systems (ICICS), pages 127–132. find
  11. Ramy Eskander, Mohamed Al-Badrashiny, Nizar Habash, and Owen Rambow. 2014. Foreign words and the automatic processing of Arabic social media text written in roman script. EMNLP 2014, page 1.
  12. Elisabetta Fersini, Paolo Rosso, and Maria Anzovino. 2018. Overview of the task on automatic misogyny identification at ibereval 2018. In IberEval@SEPLN. find
  13. Chayma Fourati, Abir Messaoudi, and Hatem Haddad. 2020. TUNIZI: a Tunisian Arabizi sentiment analysis Dataset. CoRR, abs/2004.14303. ↗
  14. Guillaume Gadek, Josefin Betsholtz, Alexandre Pauchet, Stephan Brunessaux, Nicolas Malandain, and Laurent Vercouter. 2017. Extracting contextonyms from twitter for stance detection. In Proceedings of ICAART17, pages 132–141. find
  15. Abigail S Gertner, John Henderson, Elizabeth Merkhofer, Amy Marsh, Ben Wellner, and Guido Zarrella. 2019. Mitre at semeval-2019 task 5: Transfer learning for multilingual hate speech detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 453–459. find
  16. Maria Giatsoglou, Manolis G Vozalis, Konstantinos Diamantaras, Athena Vakali, George Sarigiannidis, and Konstantinos Ch Chatzisavvas. 2017. Sentiment analysis leveraging emotions and word embeddings. Expert Systems with Applications, 69:214–224. find
  17. Yves Gonzalez-Quijano. 2014. Technology literacies of the new media: Phrasing the world in the “arab easy” (r)evolution. In Leila Hudson, Adel Iskandar, and Mimi Kirk, editors, Media Evolution on the Eve of the Arab Spring, pages 159–166. Palgrave Macmillan, New York. ↗
  18. Imane Guellil, Ahsan Adeel, Faical Azouaou, Fodil Benali, Ala-eddine Hachani, and Amir Hussain. 2018. Arabizi sentiment analysis based on transliteration and automatic corpus annotation. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 335–341, Brussels, Belgium, October. Association for Computational Linguistics. ↗
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  20. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. CoRR, abs/1907.10529. ↗
  21. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of Tricks for Efficient Text Classification. arXiv:1607.01759 [cs], August. arXiv: 1607.01759. ↗
  22. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. CoRR, abs/1901.07291. ↗
  23. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations.
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  25. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online, July. Association for Computational Linguistics.
  26. Diana Maynard and Adam Funk. 2011. Automatic detection of political opinions in tweets. In The semantic web: ESWC 2011 workshops, pages 88–99. Springer. find
  27. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 [cs], September. arXiv: 1301.3781. ↗
  28. Martin Müller, Marcel Salathé, and Per E Kummervold. 2020. Covid-twitter-bert: A natural language processing model to analyse covid-19 content on twitter. ↗
  29. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. ASTD: Arabic Sentiment Tweets Dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519, Lisbon, Portugal, sep. Association for Computational Linguistics. ↗
  30. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  31. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv:1802.05365 [cs], February. arXiv: 1802.05365. ↗
  32. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In Anna Korhonen, David R. Traum, and Lluı́s Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4996–5001. Association for Computational Linguistics.
  33. Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, and others. 1995. Okapi at TREC-3. Nist Special Publication Sp, 109:109.
  34. Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. SemEval-2015 task 10: Sentiment analysis in twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 451–463, Denver, Colorado, June. Association for Computational Linguistics.
  35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
  36. Taha Tobaili, Miriam Fernandez, Harith Alani, Sanaa Sharafeddine, Hazem Hajj, and Goran Glavaš. 2019. SenZi: A sentiment analysis lexicon for the latinised Arabic (Arabizi). In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1203–1211, Varna, Bulgaria, September. INCOMA Ltd.
  37. Taha Tobaili. 2016. Arabizi Identification in Twitter Data. In Proceedings of the ACL 2016 Student Research Workshop, pages 51–57, Berlin, Germany, aug. Association for Computational Linguistics.
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998–6008.
  39. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baert-2020-arabizi-language-models-sentiment-analysis/