Elamur Research⌕
☰
home / papers / 2020 / Combining BERT with Static Word Embeddings for Categorizing Social Media

Combining BERT with Static Word Embeddings for Categorizing Social Media

Paper · 2020 · Israa Alghanmi, Luis Espinosa Anke and Steven Schockaert

Summary

Pre-trained language models such as BERT perform well on social media classification, even though their training data (Wikipedia, news) looks very different from tweets. AraBERT is a striking case: it handles posts written in Arabic dialects despite being trained mostly on Modern Standard Arabic. The authors test whether adding static word embeddings can still improve these models. Their classifier works at the sentence level and has two branches:

  • BERT branch: the contextual vectors of a fine-tuned BERT are averaged into one sentence vector.
  • Static branch: a CNN or LSTM reads static word vectors and produces a second sentence vector.

The two vectors are concatenated, passed through dropout and fed to a softmax layer. BERT is fine-tuned first and then frozen while the rest is trained.

Across four Arabic and four English tweet benchmarks, the hybrid usually beats the fine-tuned LM by roughly one F1 point. Embeddings trained on Wikipedia help about as much as Twitter-trained ones. The gain therefore seems to come from complementary lexical information, not from domain match. This matters for low-resource languages and fast-changing vocabulary, where retraining an LM is costly.

Key points

  • Models and embeddings: AraBERTv0.1 and BERT base uncased, combined with AraVec CBOW vectors (300-d, Twitter and Wikipedia versions) and GloVe vectors (100-d, Twitter and Wikipedia versions).
  • Arabic data:
    • AJGT: 1,800 tweets
    • SemEval-2017 Task 4 Arabic subtask A: 10,126 tweets
    • L-HSAB: 5,846 tweets
    • ArsenTD-Lev: 4,000 tweets
  • English data:
    • SemEval-2018 Task 3 irony: 4,618 tweets
    • OffensEval 2019: 14,100 tweets
    • SemEval-2016 Task 6 climate stance: 564 tweets
    • SemEval-2019 Task 5 hate speech: 13,000 tweets
  • Typical gains (F1):
    • SemEval-2017 Arabic: 63.1 → 64.3
    • L-HSAB: 71.0 → 72.1
    • Irony: 67.3 → 68.4
    • Stance: 52.9 → 54.6
  • Exceptions:
    • On the English Hate set, the hybrid falls below BERT (50.6 → about 48). The authors blame overfitting caused by a train/test mismatch in annotation and tweet length.
    • On ArsenTD-Lev, only the LSTM variant helps (52.2 vs 51.8). The CNN variant drops to 49.2.
  • Evaluation setup: the BERT baseline is the best of three runs. Hybrids are averaged over five runs with min/max reported, which makes the comparison conservative toward the baseline.

Interesting ideas

  • I1. Fusing at the sentence level (averaged BERT vectors concatenated with a CNN/LSTM summary of static vectors) avoids aligning wordpieces to words, so two tokenization schemes can coexist without a merging network.
  • I2. Training in two stages (fine-tune the big model, freeze it, then train only the small add-on and the joint classifier) was more robust to overfitting than joint training on small data.
  • I3. Comparing in-domain (Twitter) embeddings with out-of-domain (Wikipedia) ones acted as a control. It showed that the improvement was not caused by the assumed domain effect.
  • I4. Static vectors are cheap to retrain, so they can cover rare and emerging words (such as trending hashtags) that a frozen LM handles poorly.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): a speech-act or irony classifier in prag could combine a transformer sentence vector with features built from our own word-level analysis, without forcing the two tokenizations to agree.
  • I2 (learn): when adding a new feature branch on top of gparse, we can freeze the trained parser and fit only the new head, so silver-data noise cannot damage the base model.
  • I3 (learn): every claimed gain from "domain-specific" resources should be tested against a generic-resource ablation, in the spirit of our negative controls.
  • I4 (tokens): the tagger and lemmatizer in crate en could use a regularly refreshed static lexicon as a fallback for unseen or rare forms, instead of retraining the whole model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cardiff UniversityCardiff · United Kingdom22 works · 2,444 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Israa Alghanmi, Luis Espinosa Anke and Steven Schockaert. 2020. Combining BERT with Static Word Embeddings for Categorizing Social Media. Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wnut-1.5

@inproceedings{alghanmi2020combining,
  title = {{Combining BERT with Static Word Embeddings for Categorizing Social Media}},
  author = {Israa Alghanmi, Luis Espinosa Anke and Steven Schockaert},
  booktitle = {Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)},
  year = {2020},
  doi = {10.18653/v1/2020.wnut-1.5},
  url = {https://doi.org/10.18653/v1/2020.wnut-1.5}
}

References

27 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Soufien Jaffali, Salma Jamoussi, Nesrine Khelifi, and Abdelmajid Ben Hamadou. 2020. Survey on social networks data analysis. In International Conference on Innovations for Community Services, pages 100– 119. Springer. find
  2. Muath Alali, Nurfadhlina Mohd Sharef, Masrah Azrifah Azmi Murad, Hazlina Hamdan, and Nor Azura Husin. 2019. Narrow convolutional neural network for arabic dialects polarity classification. IEEE Access, 7:96272–96283. ↗
  3. Ibrahim Kaibi, Hassan Satori, et al. 2020. Sentiment analysis approach based on combination of word embedding techniques. In Embedded Systems and Artificial Intelligence, pages 805–813. Springer. ↗
  4. Khaled Mohammad Alomari, Hatem M ElSherif, and Khaled Shaalan. 2017. Arabic tweets sentimental analysis using machine learning. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pages 602–610. Springer. find
  5. Bill Yuchen Lin et al. 2019. KagNet: Knowledge-aware graph networks for commonsense reasoning. In EMNLP. ↗
  6. Shoayee Alotaibi, Rashid Mehmood, and Iyad Katib. 2019. Sentiment analysis of arabic tweets in smart cities: A review of saudi dialect. In 2019 Fourth International Conference on Fog and Mobile Edge Computing (FMEC), pages 330–335. IEEE. find
  7. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. ↗
  9. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41.
  10. Ramy Baly, Gilbert Badaro, Georges El-Khoury, Rawan Moukalled, Rita Aoun, Hazem Hajj, Wassim El-Hajj, Nizar Habash, and Khaled Shaban. 2017. A characterization study of arabic twitter data with a benchmarking for state-of-the-art opinion mining models. In Proceedings of the third Arabic natural language processing workshop, pages 110–118.
  11. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshabani. 2019. L-hsab: A levantine twitter dataset for hate speech and abusive language. In Proceedings of the Third Workshop on Abusive Language Online, pages 111–118.
  12. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. Bertweet: A pre-trained language model for english tweets. arXiv preprint arXiv:2005.10200. ↗
  13. Ramy Baly, Alaa Khaddaj, Hazem Hajj, Wassim El-Hajj, and Khaled Bashir Shaban. 2019. Arsentd-lev: A multi-topic corpus for target-based sentiment analysis in arabic levantine tweets. arXiv preprint arXiv:1906.01830. ↗
  14. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  15. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Nozza Debora, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, Manuela Sanguinetti, et al. 2019. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In 13th International Workshop on Semantic Evaluation, pages 54–63. Association for Computational Linguistics.
  16. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2227–2237.
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  18. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2019. E-bert: Efficient-yet-effective entity embeddings for bert. find
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186.
  20. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval ’17, Vancouver, Canada. Association for Computational Linguistics.
  21. Timo Schick and Hinrich Schütze. 2020. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 8766–8774. ↗
  22. Maha Heikal, Marwan Torki, and Nagwa El-Makky. 2018. Sentiment analysis of arabic tweets using deep learning. Procedia Computer Science, 142:114–122. find
  23. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  24. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50.
  25. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
  26. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983. ↗
  27. Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020. Semantics-aware BERT for language understanding. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 9628–9635. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alghanmi-2020-combining-bert-static-word-embeddings/