Elamur Research
☰
home / papers / 2020 / BERTweet: A pre-trained language model for English Tweets

BERTweet: A pre-trained language model for English Tweets

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2005.10200

Summary

Popular pre-trained encoders such as BERT and RoBERTa learn mostly from edited text like Wikipedia, news and books. Tweets look very different: they are short, loosely written, and full of hashtags, abbreviations and misspellings. To close this domain gap, the authors built BERTweet, which they present as the first large public language model trained only on English Tweets. It uses the BERT-base architecture with the RoBERTa pre-training recipe, trained on about 850M English Tweets (80GB). Most of these came from an Archive Team Twitter stream, plus a small COVID-19 collection. The authors tested it on POS tagging, NER and text classification (sentiment and irony). BERTweet beat RoBERTa-base and XLM-R-base on nearly every benchmark and set new state-of-the-art results, even though those baselines saw more English data. The experiments also showed that dictionary-based lexical normalization of Tweets usually lowers scores. The model is released under the MIT License for fairseq and transformers, giving social-media NLP a strong ready-made baseline.

Key points

  • Pre-training corpus: 845M Tweets from the Archive Team Twitter Stream (01/2012–08/2019) plus 5M COVID-19 Tweets (01–03/2020), about 16B word tokens. Each Tweet has 10–64 tokens. Preprocessing used fastText language ID and the NLTK TweetTokenizer. User mentions and links became @USER and HTTPURL, and emojis were converted to text. Subwords come from fastBPE with a 64K vocabulary.
  • Training setup: RoBERTa implementation in fairseq, max sequence length 128, batch size 7K, 8 × V100 GPUs. Training ran 40 epochs, about 950K steps, over roughly 4 weeks.
  • Benchmarks and gains: Ritter11-T-POS, ARK-Twitter, TWEEBANK-V2 (POS); WNUT16, WNUT17 (NER); SemEval2017-Task4A, SemEval2018-Task3A (classification). It beat the previous SOTA by over 14 absolute points on WNUT17, about 5 points on SemEval2017-Task4A and about 4 points on SemEval2018-Task3A. The only loss to a base model was Ritter11-T-POS, where XLM-R-base scored 90.4 against BERTweet's 90.1.
  • Comparison with larger models: RoBERTa-large and XLM-R-large still scored higher on POS and NER. BERTweet beat both of them on the two classification datasets.
  • Resources: The code and model are at github.com/VinAIResearch/BERTweet. Later, the authors trained it further on 23M COVID-19 Tweets, releasing cased and uncased BERTweet-COVID19 variants.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Named entity recognition

Cited by

296 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 294 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.10200 · arXiv 2005.10200

@misc{nguyen2020bertweet,
  title = {{BERTweet: A pre-trained language model for English Tweets}},
  author = {Nguyen, Dat Quoc and Vu, Thanh and Nguyen, Anh Tuan},
  year = {2020},
  doi = {10.48550/arxiv.2005.10200},
  eprint = {2005.10200},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.10200}
}

References

42 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gustavo Aguilar, Suraj Maharjan, Adrian Pastor López-Monroy, and Thamar Solorio. 2017. A Multi-task Approach for Named Entity Recognition in Social Media Data. In Proceedings of WNUT, pages 148–153. find
  2. Eiji Aramaki. 2010. http://luululu.com/tweet/. TYPO CORPUS. ↗
  3. Christos Baziotis, Athanasiou Nikolaos, Pinelopi Papalampidi, Athanasia Kolovou, Georgios Paraskevopoulos, Nikolaos Ellinas, and Alexandros Potamianos. 2018. NTUA-SLP at SemEval-2018 task 3: Tracking ironic tweets using ensembles of word and character level attentive RNNs. In Proceedings of SemEval, pages 613–621. find
  4. Christos Baziotis, Nikos Pelekis, and Christos Doulk-eridis. 2017. DataStories at SemEval-2017 task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis. In Proceedings of SemEval, pages 747–754. find
  5. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of EMNLP-IJCNLP, pages 3615– 3620. find
  6. Steven Bird, Ewan Klein, and Edward Loper, editors. 2009. Natural language processing with Python. O’Reilly. find
  7. Mathieu Cliche. 2017. BB twtr at SemEval-2017 task 4: Twitter sentiment analysis with CNNs and LSTMs. In Proceedings of SemEval, pages 573– 580. find
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of ACL, page to appear. find
  9. Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition. In Proceedings of WNUT, pages 140–147. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171– 4186. find
  11. Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of NAACL-HLT, pages 359–369. find
  12. Norjihan Abdul Ghani, Suraya Hamid, Ibrahim Abaker Targio Hashem, and Ejaz Ahmed. 2019. Social media big data analytics: A survey. Comput. Hum. Behav., 101:417–428. find
  13. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments. In Proceedings of ACL-HLT, pages 42–47. find
  14. Tao Gui, Qi Zhang, Jingjing Gong, Minlong Peng, Di Liang, Keyu Ding, and Xuanjing Huang. 2018. Transferring from Formal Newswire Domain with Hypernet for Twitter POS Tagging. In Proceedings of EMNLP, pages 2540–2549. find
  15. Tao Gui, Qi Zhang, Haoran Huang, Minlong Peng, and Xuanjing Huang. 2017. Part-of-Speech Tagging for Twitter with Adversarial Neural Networks. In Proceedings of EMNLP, pages 2411–2420. find
  16. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of ACL, pages 8342–8360. find
  17. Bo Han, Paul Cook, and Timothy Baldwin. 2012. Automatically Constructing a Normalisation Dictionary for Microblogs. In Proceedings of EMNLP-CoNLL, pages 421–432. find
  18. Bo Han, Paul Cook, and Timothy Baldwin. 2013. Lexical Normalization for Social Media Text. ACM Transactions on Intelligent Systems and Technology, 4(1). find
  19. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of EACL, pages 427–431. find
  20. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint, arXiv:1412.6980. ↗
  21. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, page btz682. find our card
  22. Nut Limsopatham and Nigel Collier. 2016. Bidirectional LSTM for Named Entity Recognition in Twitter Messages. In Proceedings of WNUT, pages 145– 152. find
  23. Fei Liu, Fuliang Weng, and Xiao Jiang. 2012. A Broad-Coverage Normalization System for Social Media Language. In Proceedings of ACL, pages 1035–1044. find
  24. Yijia Liu, Yi Zhu, Wanxiang Che, Bing Qin, Nathan Schneider, and Noah A. Smith. 2018. Parsing Tweets into Universal Dependencies. In Proceedings of NAACL-HLT, pages 965–975. find our card
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint, arXiv:1907.11692. ↗
  26. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In Proceedings of ICLR. find
  27. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of ACL, pages 1064–1074. find our card
  28. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. ↗ our card
  29. Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020. WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. In Proceedings of WNUT. find
  30. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, pages 48–53. find our card
  31. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. 2013. Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters. In Proceedings of NAACL-HLT, pages 380–390. find our card
  32. Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named Entity Recognition in Tweets: An Experimental Study. In Proceedings of EMNLP, pages 1524–1534. find
  33. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 Task 4: Sentiment Analysis in Twitter. In Proceedings of SemEval, pages 502–518. find our card
  34. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of ACL, pages 1715–1725. find
  35. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A gold standard dependency corpus for English. In Proceedings of LREC. find
  36. Benjamin Strauss, Bethany Toma, Alan Ritter, Marie-Catherine de Marneffe, and Wei Xu. 2016. Results of the WNUT16 Named Entity Recognition Shared Task. In Proceedings of WNUT, pages 138–144. find
  37. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of SemEval, pages 39–50. find our card
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. find
  39. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv preprint, arXiv:1910.03771. ↗ our card
  40. Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhigang Yuan, and Yongfeng Huang. 2018. THU NGN at SemEval-2018 task 3: Tweet irony detection with densely connected LSTM and multi-task learning. In Proceedings of SemEval, pages 51– 56. find
  41. Joey Tianyi Zhou, Hao Zhang, Di Jin, Hongyuan Zhu, Meng Fang, Rick Siow Mong Goh, and Kenneth Kwok. 2019. Dual Adversarial Neural Transfer for Low-Resource Named Entity Recognition. In Proceedings of ACL, pages 3461–3471. find
  42. Shi Zong, Ashutosh Baheti, Wei Xu, and Alan Ritter. 2020. Extracting COVID-19 Events from Twitter. arXiv preprint, arXiv:2006.02567. ↗