Elamur Research⌕
☰
home / papers / 2019 / AlBERTo: Italian BERT Language Understanding Model for NLP Challenging Tasks Based on Tweets

AlBERTo: Italian BERT Language Understanding Model for NLP Challenging Tasks Based on Tweets

Summary

Multilingual pre-trained encoders tend to underperform on individual languages, especially when the text style differs from books and encyclopedias. The authors address this for Italian social media by building AlBERTo, a BERT-base model trained from scratch on Italian Twitter data. They sampled 200 million tweets, excluding retweets, from the TWITA corpus. The tweets were normalized with Ekphrasis, which replaces URLs, mentions, numbers and similar items with placeholder tags and unpacks hashtags, and were then segmented with a SentencePiece vocabulary. Pre-training used only the masked language modelling objective. Next-sentence prediction was dropped because individual tweets do not form a coherent sequence. The authors fine-tuned separate binary classifiers on the EVALITA 2016 SENTIPOLC data. AlBERTo beat the best shared-task systems on subjectivity, polarity and irony detection. The paper reports gains of roughly 7–11% over the top participants. The released model gives Italian NLP a reusable, domain-matched starting point for analysing social media text, which is one of the main application areas for the language.

Key points

  • Architecture: a BERT-base configuration with 12 layers, hidden size 768 and 12 attention heads. The SentencePiece vocabulary has 128,000 entries and the maximum sequence length is 128.
  • Pre-training data and cost: 200M TWITA tweets, about 191GB of raw data. Training ran for 1M steps with batch size 128 and a masking probability of 0.15. It took about 50 hours on an 8-core Google TPU-V2 through Colab.
  • Evaluation data: SENTIPOLC 2016 provides 7,410 training tweets and 2,000 test tweets. No validation set or model selection was used.
  • Combined F scores vs. the best shared-task system:
    • Subjectivity: 0.7906 vs. 0.7444 (Unitor.1.u)
    • Polarity: 0.7223 vs. 0.6638 (UniPI.2.c)
    • Irony: 0.6090 vs. 0.5412 (tweet2check16.c)
  • Fine-tuning setup: four classifiers were trained (subjectivity, positive polarity, negative polarity, irony), each taking about 4 minutes. Irony needed 10 epochs instead of 3, and its recall on the ironic class remained low at 0.1787.

Interesting ideas

  • I1. Match the pre-training corpus to the target genre. A monolingual model trained on the same register as the downstream data, here tweets, can outperform general-purpose systems. This argues for building, for example, a social-media-specific Ukrainian encoder rather than relying only on multilingual or formal-text models.
  • I2. Normalize noisy entities into typed placeholders. Replacing URLs, mentions, dates and numbers with tags, and splitting hashtags into words, shrinks vocabulary sparsity. The useful signal that "a link or mention is here" is still kept. This is cheap and portable to any user-generated-text pipeline.
  • I3. Drop pre-training objectives whose assumptions the data breaks. Next-sentence prediction was removed because tweets are not a continuous discourse. It is worth checking whether each auxiliary objective actually fits your corpus, with the trade-off stated explicitly: here, weaker suitability for QA-style tasks.
  • I4. Split a multi-class label into binary detectors. Training separate positive and negative classifiers lets mixed sentiment arise naturally when both fire. This is a simple design for overlapping label schemes.
  • I5. Look beyond aggregate F1 to per-class precision and recall. The per-class breakdown showed a conservative model, with high precision and low recall on the minority class, especially for irony. Analysis like this shows where threshold tuning or class rebalancing would help.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

76 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{polignano2019alberto,
  title = {{AlBERTo: Italian BERT Language Understanding Model for NLP Challenging Tasks Based on Tweets}},
  author = {Polignano, Marco and Basile, Pierpaolo and Gemmis, Marco de and Semeraro, Giovanni and Basile, Valerio},
  booktitle = {Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019)},
  year = {2019},
  url = {https://aclanthology.org/2019.clicit-1.47/}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Francesco Barbieri, Valerio Basile, Danilo Croce, Malvina Nissim, Nicole Novielli, and Viviana Patti. 2016. Overview of the evalita 2016 sentiment polarity classification task. In Proceedings of third Italian conference on computational linguistics (CLiC-it 2016) & fifth evaluation campaign of natur… find
  2. Final Workshop (EVALITA 2016). find
  3. Pierpaolo Basile, Franco Cutugno, Malvina Nissim, Viviana Patti, Rachele Sprugnoli, et al. 2016. find
  4. Evalita 2016: Overview of the 5th evaluation campaign of natural language processing and speech tools for italian. In 3rd Italian Conference on Computational Linguistics, CLiC-it 2016 and 5th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, EVALITA 2016, volume 1749,… find
  5. Valerio Basile, Mirko Lai, and Manuela Sanguinetti. 2018. Long-term social media data collection at the university of turin. In Fifth Italian Conference on Computational Linguistics (CLiC-it 2018), pages 1– 6. CEUR-WS. find
  6. Christos Baziotis, Nikos Pelekis, and Christos Doulk-eridis. 2017. Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747–754, Vancouver, Can… find our card
  7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  8. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. find our card
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  10. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959. find
  11. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  12. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  13. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. pages 2227–2237, June. find
  14. Marco Polignano, Pierpaolo Basile, Marco de Gemmis, and Giovanni Semeraro. 2019. A comparison of word-embeddings in emotion detection from text using bilstm, cnn and self-attention. In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization, pages 63–68. ACM. find
  15. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  17. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  18. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/polignano-2019-alberto-italian-bert-language-understanding/