Elamur Research⌕
☰
home / papers / 2020 / LynyrdSkynyrd at WNUT-2020 Task 2: Semi-Supervised Learning for Identification of Informative COVID-19 English Tweets

LynyrdSkynyrd at WNUT-2020 Task 2: Semi-Supervised Learning for Identification of Informative COVID-19 English Tweets

Paper · 2020 · Abhilasha Sancheti, Kushal Chawla and Gaurav Verma

Summary

The paper presents a system for the W-NUT 2020 shared task, which asks whether an English COVID-19 tweet is "informative," meaning it reports cases, deaths, recoveries, suspected infections, or the locations and travel of patients. The authors use the official COVID-19 Tweets dataset and apply heavy normalization: placeholders for URLs and usernames, expanded contractions, emojis turned into words, and a single spelling for COVID variants. They also build dataset variants in which numbers and geopolitical entities are masked. Their base classifiers range from fastText, n-gram, hand-crafted affect/syntax, and Universal Sentence Encoder models to BERT fine-tuning. The BERT models get extra masked-LM pretraining on about 27k unlabelled pandemic tweets, and the feature-based models are trained with confident pseudo-labels. A logistic-regression stacker with a tuned decision threshold combines the base models and reaches 0.9179 F1 on validation and 0.8805 on the blind test set. The authors also probe the classifier with hand-edited examples. These probes show that it relies on surface cues such as "confirmed case," digits, and place names, which is a warning for anyone deploying it in monitoring tools.

Key points

  • Data split is 7000 train, 1000 validation and 2000 blind test tweets. The metric is F1 on the Informative class.
  • BERT-IDPT-FiT uses bert-base-uncased with in-domain MLM pretraining on 27,388 collected tweets. It reaches 0.9050 F1, beating plain fine-tuning (0.8977) and in-task pretraining (0.8984).
  • In the pseudo-labelling loop, after every 10 epochs (50 at most), the best checkpoint labels the unlabelled pool. From the predictions with confidence above 0.99, 1000 are sampled and moved into training. This helped all three feature-based models (HCF, USE, ngram).
  • Feature models top out near 0.80 F1 (ngram-PL 0.7997), roughly ten points below the BERT variants. Larger encoders brought no visible gain: BERT-Large, RoBERTa, XLNet and Covid-BERT all performed about the same.
  • Ensemble-TT stacks base-model probabilities with logistic regression and uses a threshold of 0.5168. Compared with the individual models, it mainly improves precision while keeping recall.

Interesting ideas

  • I1. Minimal-edit probing: swap single tokens ("coronavirus" vs "malaria," a frequent vs a rare city, a word vs a digit) and watch whether the label flips, exposing shortcut cues cheaply.
  • I2. Masking numbers as NUM plus a digit count, and GPE entities as LOC, keeps the magnitude signal and the fact that a place is mentioned, while stopping the model from memorizing particular values or locations.
  • I3. Self-training with strict acceptance rules works better than bulk self-labelling: only very confident predictions are used, they come in small random batches, and training always resumes from the best checkpoint.
  • I4. Stacking heterogeneous models through a small logistic regression and then tuning its decision threshold is a cheap and transparent way to combine weak and strong signals.
  • I5. Continued masked-LM pretraining on unlabelled text from the target domain beat pseudo-labelling for the transformer model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build negative-control suites of minimally edited sentences and check that the frames or answers change only when the meaning actually changes.
  • I2 (tokens): Optional placeholder normalization of numerals and place names could make training of the tagger and parser less tied to specific vocabulary in noisy text.
  • I3 (learn): The same caps and confidence gates could control how much silver data that witness parsers agree on enters each training round of the perceptron parser.
  • I4 (tree): A logistic regression over witness-parser agreement features could produce an interpretable confidence score for each arc or each sentence.
  • I5 (tree): Adapt gparse to new genres such as tweets or fables with domain MLM pretraining before treebank fine-tuning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States162 works · 35,836 citations🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations🏢Adobe SystemsSan Jose · United States35 works · 3,711 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Abhilasha Sancheti, Kushal Chawla and Gaurav Verma. 2020. LynyrdSkynyrd at WNUT-2020 Task 2: Semi-Supervised Learning for Identification of Informative COVID-19 English Tweets. Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wnut-1.65

@inproceedings{sancheti2020lynyrdskynyrd,
  title = {{LynyrdSkynyrd at WNUT-2020 Task 2: Semi-Supervised Learning for Identification of Informative COVID-19 English Tweets}},
  author = {Abhilasha Sancheti, Kushal Chawla and Gaurav Verma},
  booktitle = {Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)},
  year = {2020},
  doi = {10.18653/v1/2020.wnut-1.65},
  url = {https://doi.org/10.18653/v1/2020.wnut-1.65}
}

References

16 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julian Brooke, Tong Wang, and Graeme Hirst. 2010. Automatic acquisition of lexical formality. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters. Association for Computational Linguistics. find
  2. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  3. Armand Joulin, Édouard Grave, Piotr Bojanowski, and Tomáš Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. find
  4. Rabindra Lamsal. 2020. tweets dataset. Coronavirus (covid-19) find
  5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  6. Christian E Lopez, Malolan Vasu, and Caleb Gallemore. 2020. Understanding the perception of covid-19 policies by mining a multilanguage twitter dataset. arXiv preprint arXiv:2003.10359. ↗
  7. Saif Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). ↗
  8. Martin Müller, Marcel Salathé, and Per E Kummervold. 2020. Covid-twitter-bert: A natural language processing model to analyse covid-19 content on twitter. arXiv preprint arXiv:2005.07503. ↗
  9. Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020. WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. In Proceedings of the 6th Workshop on Noisy User-generated Text.
  10. Gregory Park, H Andrew Schwartz, Maarten Sap, Margaret L Kern, Evan Weingarten, Johannes C Eichstaedt, Jonah Berger, David J Stillwell, Michal Kosinski, Lyle H Ungar, et al. 2015. Living in the past, present, and future: Measuring temporal orientation with language. Journal of personality. ↗
  11. James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates. find
  12. Martin EP Seligman. 2012. Flourish: A visionary new understanding of happiness and well-being. Simon and Schuster. find
  13. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In China National Conference on Chinese Computational Linguistics. Springer. ↗
  14. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 english lemmas. Behavior research methods. ↗
  15. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems. ↗
  16. Shi Zong, Ashutosh Baheti, Wei Xu, and Alan Ritter. 2020. Extracting covid-19 events from twitter. arXiv preprint arXiv:2006.02567. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sancheti-2020-lynyrdskynyrd-wnut-2020-task-2/