Elamur Research⌕
☰
home / papers / 2022 / XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond

XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond

Summary

Multilingual language models are usually built and evaluated on clean text such as Wikipedia or curated benchmarks. Social media, and Twitter in particular, has received little attention, even though tweets are full of slang, misspellings, emoji and mixed signals. The authors address this gap with XLM-T, a toolkit for working with Twitter-oriented multilingual models. They took the public XLM-R base checkpoint and continued masked-LM training on 198M tweets in more than thirty languages, producing XLM-Twitter. They also assembled UMSAB, a unified sentiment benchmark covering eight languages. Python starter code covers embeddings, fine-tuning, inference and evaluation.

On English TweetEval, XLM-Twitter outperforms general-domain XLM-R and RoBERTa-base. It still lags behind English-only Twitter models. On UMSAB, it beats XLM-R in most monolingual settings and in the joint multilingual setting. Results for low-performing languages such as Hindi suggest that well-chosen transfer data can matter more than simply adding many languages. The released models and data give the community a reusable baseline for multilingual social-media NLP.

Key points

  • Pretraining corpus: 198M tweets collected between May 2018 and March 2020, about 1,724M tokens (12G of text).
    • Tweets were kept only if they had at least three tokens and no URLs, to reduce bots and spam.
    • No language filtering was applied, and Ukrainian appears among the languages in the data.
  • Training cost: Continued pretraining of xlm-roberta-base took about 14 days on 8 NVIDIA V100 GPUs, with an estimated cost of roughly USD 5,000.
  • UMSAB benchmark: Covers Arabic, English, French, German, Hindi, Italian, Portuguese and Spanish.
    • Scores are macro-F1 averaged over three seeds.
    • Only the learning rate was tuned (0.001 vs 0.0001).
  • Multilingual results: When fine-tuned on all languages together, XLM-Twitter reaches 69.45 macro-F1 on the combined test set, against 66.93 for XLM-R.
  • Hindi transfer: Hindi with XLM-Twitter scores about 40.3 when trained monolingually. It rises to about 55.4 when the adapter is first trained on Italian, the best zero-shot source.

Interesting ideas

  • I1. Continue pretraining instead of starting from scratch. Further masked-LM training of an existing multilingual checkpoint on in-domain text is a relatively cheap way to obtain a domain-adapted model. It could suit Ukrainian social-media or news corpora where training from scratch is unaffordable.
  • I2. Filter with simple heuristics only. Dropping very short posts and posts with links is an easy way to remove much bot and advertising noise. Skipping language filtering preserves the platform's natural language mix, including code-switching.
  • I3. Use adapters for fine-tuning. The base model stays frozen and only small extra parameters are trained. This saves memory and time, which makes multilingual models practical for teams with modest hardware.
  • I4. Pick transfer languages on validation data. For a weak target language, first find the source language that gives the best zero-shot score on validation data. Then initialize the target adapter from that source. This targeted transfer can beat a naive monolingual setup and could help low-resource Ukrainian tasks.
  • I5. Check monolingual cost alongside multilingual gains. Scoring the multilingual model on an established English benchmark shows how much it loses against dedicated monolingual models. This helps decide when a language-specific model is worth building.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

106 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 105 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{barbieri2022xlm,
  title = {{XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond}},
  author = {Barbieri, Francesco and Anke, Luis Espinosa and Camacho-Collados, Jose},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/4fhqtpvii3vg},
  url = {https://doi.org/10.63317/4fhqtpvii3vg}
}

References

45 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Barbieri, F., Basile, V., Croce, D., Nissim, M., Novielli, N., and Patti, V. (2016). Overview of the evalita 2016 sentiment polarity classification task. In Proceedings of third Italian conference on computational linguistics (CLiC-it 2016) & fifth evaluation campaign of natural language processing… find
  2. Final Workshop (EVALITA 2016). find
  3. Barbieri, F., Camacho-Collados, J., Ronzano, F., Anke, L. E., Ballesteros, M., Basile, V., Patti, V., and Saggion, H. (2018). find
  4. Semeval 2018 task 2: Multilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 24–33. find
  5. Barbieri, F., Camacho-Collados, J., Espinosa Anke, L., and Neves, L. (2020). TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650, Online, November. Association for Computational Li… find our card
  6. Barnes, J., Øvrelid, L., and Velldal, E. (2019). Sentiment analysis is not solved! assessing and probing sentiment classification. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 12–23, Florence, Italy, August. Association for Computatio… find
  7. Barriere, V. and Balahur, A. (2020). Improving sentiment analysis over non-English tweets using multilingual transformers and automatic translation for data-augmentation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 266–271, Barcelona, Spain (Online), Decem… find
  8. Basile, V., Bosco, C., Fersini, E., Nozza, D., Patti, V., Rangel Pardo, F. M., Rosso, P., and Sanguinetti, M. (2019). SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54… find
  9. Benamara, F., Grouin, C., Karoui, J., Moriceau, V., and Robba, I. (2017). Analyse d’opinion et langage figuratif dans des tweets: présentation et résultats du défi fouille de textes deft2017. In Défi Fouille de Textes DEFT2017. find
  10. Atelier TALN 2017. Association pour le Traitement Automatique des Langues (ATALA). find
  11. Brum, H. B. and Nunes, M. d. G. V. (2017). Building a sentiment corpus of tweets in brazilian portuguese. arXiv preprint arXiv:1712.08917. find
  12. Camacho-Collados, J., Doval, Y., Martı́nez-Cámara, E., Espinosa-Anke, L., Barbieri, F., and Schockaert, S. (2020). Learning cross-lingual word embeddings from twitter via distant supervision. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 72–82. find
  13. Chen, X., Sun, Y., Athiwaratkun, B., Cardie, C., and Weinberger, K. (2018). Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557–570. find
  14. Chronopoulou, A., Baziotis, C., and Potamianos, A. (2019). An embarrassingly simple approach for transfer learning from pretrained language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… find
  15. Cieliebak, M., Deriu, J. M., Egger, D., and Uzdilli, F. (2017). A twitter corpus and benchmark resources for german sentiment analysis. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 45–51. find
  16. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages… find
  17. Derczynski, L., Ritter, A., Clark, S., and Bontcheva, K. (2013). Twitter part-of-speech tagging for all: Overcoming sparse and noisy data. In Proceedings of the international conference recent advances in natural language processing ranlp 2013, pages 198– 206. find
  18. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (… find
  19. Dı́az-Galiano, M. C., Martı́nez-Cámara, E., Ángel Garcı́a Cumbreras, M., Vega, M. G., and Román, J. V. (2018). The democratization of deep learning in tass 2017. Procesamiento del Lenguaje Natural, 60(0):37–44. find
  20. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018). Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). find our card
  21. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N. A. (2020). Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online, July. As… find
  22. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. find
  23. Hu, J., Ruder, S., Siddhant, A., Neubig, G., Firat, O., and Johnson, M. (2020). Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR. find
  24. Joshi, P., Santy, S., Budhiraja, A., Bali, K., and Choudhury, M. (2020). The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online, July. Association for Computational… find our card
  25. Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2017). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain, April. Association f… find
  26. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pre-training approach. arXiv preprint arXiv:1907.11692. find
  27. Meng, X., Wei, F., Liu, X., Zhou, M., Xu, G., and Wang, H. (2012). Cross-lingual mixture model for sentiment classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 572–581. Association for Computational Linguistics. find
  28. Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X., and Cherry, C. (2016). Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41. find
  29. Mohammad, S., Bravo-Marquez, F., Salameh, M., and Kiritchenko, S. (2018). Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17. find our card
  30. Nguyen, D. Q., Vu, T., and Tuan Nguyen, A. (2020). BERTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9–14, Online, October. Association for Computational Linguistics. find our card
  31. Patra, B. G., Das, D., Das, A., and Prasath, R. (2015). Shared task on sentiment analysis in indian languages (sail) tweets-an overview. In International Conference on Mining Intelligence and Knowledge Exploration, pages 650–655. Springer. find
  32. Patwa, P., Aguilar, G., Kar, S., Pandey, S., PYKL, S., Gambäck, B., Chakraborty, T., Solorio, T., and Das, A. (2020). SemEval-2020 task 9: Overview of sentiment analysis of code-mixed tweets. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 774–790, Barcelona (online), Decemb… find
  33. Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., and Gurevych, I. (2020). Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247. find our card
  34. Pires, T., Schlinger, E., and Garrette, D. (2019). How multilingual is multilingual bert? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001. find our card
  35. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67. find
  36. Rasooli, M. S., Farra, N., Radeva, A., Yu, T., and McKeown, K. (2018). Cross-lingual sentiment transfer with limited resources. Machine Translation, 32(1):143–165, Jun. find
  37. Rosenthal, S., Farra, N., and Nakov, P. (2017). Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pages 502–518. find our card
  38. Rosenthal, S., Farra, N., and Nakov, P. (2019). Semeval-2017 task 4: Sentiment analysis in twitter. arXiv preprint arXiv:1912.00741. find our card
  39. Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., and Gurevych, I. (2020). How good is your tokenizer? on the monolingual performance of multilingual language models. arXiv preprint arXiv:2012.15613. find our card
  40. Salameh, M., Mohammad, S., and Kiritchenko, S. (2015). Sentiment after translation: A case-study on arabic social media posts. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 767–777, Denver, Col… find
  41. Van Hee, C., Lefever, E., and Hoste, V. (2018). Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39–50. find our card
  42. Vilares, D., Alonso, M. A., and Gómez-Rodrı́guez, C. (2017). Supervised sentiment analysis in multilingual environments. Information Processing & Management, 53(3):595 – 607. find
  43. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mt5: A massively multilingual pre-trained text-to-text transformer. find
  44. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019). SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis, Minnesota, USA,… find our card
  45. Zhou, X., Wan, X., and Xiao, J. (2016). Cross-lingual sentiment classification with bilingual document representation learning. In Proceedings of ACL, pages 1403–1412, August. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/barbieri-2022-xlm-t-multilingual-language-models/