Elamur Research⌕
☰
home / papers / 2020 / NLP North at WNUT-2020 Task 2: Pre-training versus Ensembling for Detection of Informative COVID-19 English Tweets

NLP North at WNUT-2020 Task 2: Pre-training versus Ensembling for Detection of Informative COVID-19 English Tweets

Paper · 2020 · Anders Giovanni Møller, Rob van der Goot and Barbara Plank

Summary

This paper describes a system for WNUT-2020 Task 2, a binary task that sorts English COVID-19 tweets into INFORMATIVE and UNINFORMATIVE. A tweet counts as informative when it reports things like case statistics, locations or travel history. The organisers released 7,000 training tweets, 1,000 development tweets and a test set of 12,000, of which 2,000 were annotated for scoring.

The authors compared three model families:

  • a linear SVM over sparse n-grams;
  • MLP and Conv1d networks on frozen Twitter GloVe vectors;
  • three fine-tuned transformers: BERT-base, RoBERTa-base and CT-BERT, a BERT-Large model further pre-trained on COVID tweets.

They then tried to combine these models through voting, stacking and confidence-based fallback. CT-BERT alone performed best, and no ensemble beat it. On the hidden test data it reached 90.96 F1, which tied for first place among 98 submissions. The main lesson is that pre-training on text from the target domain matters more than the sheer amount of pre-training data. Ensembles added cost without adding accuracy.

Key points

  • Development weighted F1 for single models:
    • CT-BERT 92.19, BERT-base 89.86, RoBERTa-base 89.59;
    • SVM 83.64, MLP 78.05, Conv1d 75.52.
  • The best SVM used word 1-2 grams plus character 5-6 grams. Adding wordpiece n-grams lowered the score slightly, and the n-gram search gained roughly 2%.
  • RoBERTa saw far more general text (160GB) than BERT-base but scored about the same. CT-BERT, trained on 22.5M corona tweets, was about 2.6–2.9 points higher than both.
  • Ensembles on development data:
    • soft voting 88.19;
    • hard voting 89.99;
    • stacking with a Random Forest 91.67, i.e. −0.55% relative to CT-BERT;
    • SVM fallback on low CT-BERT confidence never exceeded CT-BERT alone.
  • On test data, CT-BERT scored 90.96, tied with team NutCracker for first of 98 submissions from 55 teams. The stacking ensemble scored 90.54.

Interesting ideas

  • I1. Pre-training on domain-matched text of moderate size beat a much larger generic corpus, so the choice of pre-training data deserves effort before scale does.
  • I2. Let a very different system take over when the main model's confidence is low, sweep the threshold, and plot the result against a flat line for the main model alone, which makes "no gain" plain to see.
  • I3. A stacking meta-classifier can take both each model's label and its class probabilities, and a random forest handles mixed binary and continuous inputs plus their interactions.
  • I4. Tune each feature granularity separately (words, characters, wordpieces) over n = 1–7, then remove one group at a time to confirm what actually helps.
  • I5. Report a cheap model that needs no pre-training next to the heavy ones, because it is far faster and may be good enough in some settings.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Continue pre-training the gparse encoder on fables, tales and school-course text instead of only on generic corpora, and measure LAS against the generic encoder.
  • I2 (learn): Use the same sweep to decide when a perceptron witness should override gparse on low-confidence arcs, keeping the standalone parser as the baseline line in the plot.
  • I3 (learn): Feed witness-parser votes and confidences into a small stacker to decide which silver arcs to accept, rather than relying on plain majority agreement.
  • I4 (tokens): Apply drop-one-group ablation to the tagger's feature templates so the crate keeps only the feature families that earn their place.
  • I5 (answer): Pair each reader benchmark with a simple n-gram baseline so gains from the explainable pipeline are measured against a meaningful floor.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓IT University of CopenhagenCopenhagen · Denmark27 works · 2,406 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Anders Giovanni Møller, Rob van der Goot and Barbara Plank. 2020. NLP North at WNUT-2020 Task 2: Pre-training versus Ensembling for Detection of Informative COVID-19 English Tweets. Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wnut-1.44

@inproceedings{moller2020nlp,
  title = {{NLP North at WNUT-2020 Task 2: Pre-training versus Ensembling for Detection of Informative COVID-19 English Tweets}},
  author = {Anders Giovanni Møller, Rob {van der Goot} and Barbara Plank},
  booktitle = {Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)},
  year = {2020},
  doi = {10.18653/v1/2020.wnut-1.44},
  url = {https://doi.org/10.18653/v1/2020.wnut-1.44}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Angelo Basile, Gareth Dwyer, Maria Medvedeva, Josine Rawee, Hessel Haagsma, and Malvina Nissim. 2018. Simply the best: Minimalist system trumps complex models in author profiling. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 143–156, Cham. Springer International Publishing. find
  2. Leo Breiman. 2001. Random forests. Machine learning, 45(1):5–32. find
  3. Çağrı Çöltekin and Taraka Rama. 2018. Tübingen-oslo at SemEval-2018 task 2: SVMs perform better than RNNs in emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 34–38, New Orleans, Louisiana. Association for Computational Linguistics. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus . Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pretraining for biomedical natural language processing. arXiv preprint arXiv:2007.15779. ↗
  6. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. ↗
  7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  8. Ilya Loshchilov and Frank Hutter. 2017. weight decay regularization in adam. abs/1711.05101. ↗
  9. Fixing CoRR, Christopher D. Manning. 2015. Computational linguistics and deep learning. Computational Linguistics, 41(4):701–707. ↗
  10. Maria Medvedeva, Martin Kroon, and Barbara Plank. 2017. When sparse traditional models outperform dense neural networks: the curious case of discriminating between similar languages. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 156– 163, Valencia, Spain. Association for Computational Linguistics. find
  11. M. Müller, Marcel Salathé, and P. Kummervold. 2020. Covid-twitter-bert: A natural language processing model to analyse covid-19 content on twitter. ArXiv, abs/2005.07503. ↗
  12. Sebastian Nagel. 2016. https://commoncrawl.org/2016/10/news-dataset-available/ . ↗
  13. Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020. WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. In Proceedings of the 6th Workshop on Noisy User-generated Text.
  14. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  15. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. ↗
  16. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  17. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. IEEE. ↗
  18. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. CoRR, abs/1806.02847. ↗
  19. David Wolpert. 1992. Stacked generalization. Neural Networks, 5:241–259. find
  20. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the VarDial evaluation campaign 2017. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 1– 15, Valencia, Spain. Association for Computational Linguistics.
  21. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. CoRR, abs/1506.06724. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/moller-2020-nlp-north-wnut-2020-task/