Elamur Research⌕
☰
home / papers / 2020 / NIT_COVID-19 at WNUT-2020 Task 2: Deep Learning Model RoBERTa for Identify Informative COVID-19 English Tweets

NIT_COVID-19 at WNUT-2020 Task 2: Deep Learning Model RoBERTa for Identify Informative COVID-19 English Tweets

Paper · 2020 · Jagadeesh M S and Alphonse P J A

Summary

The paper describes the NIT COVID-19 team's entry to WNUT-2020 Task 2, a shared task on noisy social media text. The task is to decide whether an English tweet about the pandemic is INFORMATIVE or UNINFORMATIVE. A tweet counts as informative if it reports concrete facts such as suspected, confirmed, recovered or fatal cases, or where patients were located or travelled. The authors clean the tweets in a classic pipeline: nltk tokenization, lowercasing, punctuation and stop-word removal, and stemming. They then fine-tune roberta-base on the organizers' COVID-19 tweet dataset (Nguyen et al., 2020). They compare it with Random Forest, SVM, CNN and BERT baselines. RoBERTa came out on top, though only narrowly ahead of BERT. Its official test F1 was 89.14. The work matters as a practical reference point: an off-the-shelf pretrained transformer handled a crisis-monitoring filter well, and classical feature-based models trailed by roughly six to eight F1 points.

Key points

  • Data: 7,000 training and 1,000 validation tweets, plus 12,000 test tweets in which only 2,000 hidden ones (944 INFORMATIVE / 1,056 UNINFORMATIVE) were scored.

  • RoBERTa setup:

    • roberta-base (hidden size 768) with Adam and a maximum learning rate of 1e-5
    • batch size 16 and maximum sequence length 143
    • hidden dropout 0.05, trained for 50 epochs

  • Validation F1 by model:

    ModelF1
    Random Forest81.59
    SVM (1–3-gram TF-IDF)82.71
    CNN83.74
    BERT88.98
    RoBERTa89.19

  • Official test results: F1 89.14, accuracy 89.35 and precision 85.94. Recall is given as 92.58 in the table but 92.35 in the text. The final model was retrained on the training and validation sets combined.

  • On validation, per-class F1 was 89.89 for INFORMATIVE and 88.16 for UNINFORMATIVE. The paper's error discussion is internally inconsistent about which class causes most mistakes.

Interesting ideas

  • I1. The organizers buried the 2,000 scored test tweets inside a 12,000-tweet pool, so participants could not tell which items counted and could not hand-tune on them.
  • I2. The paper climbs a ladder of baselines from Random Forest through SVM and CNN to BERT and RoBERTa, showing the jump from sparse features to pretrained encoders and the tiny gap between the two transformers.
  • I3. After choosing hyperparameters on validation, the authors refit on training plus validation to get more data for the final run.
  • I4. "Informative" is defined by content types rather than tone: case counts, outcomes, locations and travel history.
  • I5. The model's recall is noticeably higher than its precision, and the authors inspect per-class scores and a confusion matrix rather than relying on one aggregate number.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation sets could hide the graded items among many unscored distractors, which acts as a negative control against overfitting to a known test list.
  • I2 (tree): Comparing the perceptron parser and gparse side by side on the same data would show where the GPU transformer really pays off and where the cheaper model is nearly as good.
  • I3 (learn): Once a configuration is frozen, folding the held-out silver or gold data back into training is a cheap gain, as long as an independent test set is kept aside.
  • I4 (world): A text could be flagged as fact-reporting when its world model contains entities with numeric states and places or movements, giving an explainable alternative to a black-box classifier.
  • I5 (answer): Reader evaluations should report per-question-type breakdowns and error directions, so systematic over-answering or under-answering becomes visible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Jagadeesh M S and Alphonse P J A. 2020. NIT_COVID-19 at WNUT-2020 Task 2: Deep Learning Model RoBERTa for Identify Informative COVID-19 English Tweets. Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wnut-1.66

@inproceedings{s2020nit,
  title = {{NIT\_COVID-19 at WNUT-2020 Task 2: Deep Learning Model RoBERTa for Identify Informative COVID-19 English Tweets}},
  author = {Jagadeesh M S and Alphonse P J A},
  booktitle = {Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)},
  year = {2020},
  doi = {10.18653/v1/2020.wnut-1.66},
  url = {https://doi.org/10.18653/v1/2020.wnut-1.66}
}

References

19 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Reem ALRashdi and Simon O’Keefe. 2019. Deep learning and word embeddings for tweet classification for crisis response. arXiv preprint arXiv:1903.11024. ↗
  2. Reshma C Bhagat and Sachin S Patil. 2015. Enhanced smote algorithm for classification of imbalanced bigdata using random forest. In 2015 IEEE International Advance Computing Conference (IACC), pages 403–408. IEEE. find
  3. Aditya Bohra, Deepanshu Vijay, Vinay Singh, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018. A dataset of hindi-english code-mixed social media text for hate speech detection. In Proceedings of the second workshop on computational modeling of people’s opinions, personality, and emotions in social media, pages 36–41.
  4. Natalie M Linton, Tetsuro Kobayashi, Yichi Yang, Katsuma Hayashi, Andrei R Akhmetzhanov, Sung-mok Jung, Baoyin Yuan, Ryo Kinoshita, and Hiroshi Nishiura. 2020. Incubation period and other epidemiological characteristics of 2019 novel coronavirus infections with right truncation: a statistical analysis of publicly available case data. Journal of clinical medicine, 9(2):538. ↗
  5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  6. Christian E Lopez, Malolan Vasu, and Caleb Gallemore. 2020. Understanding the perception of covid-19 policies by mining a multilanguage twitter dataset. arXiv preprint arXiv:2003.10359. ↗
  7. Sreenivasulu Madichetty and M Sridevi. 2019. Detecting informative tweets during disaster using deep neural networks. In 2019 11th International Conference on Communication Systems & Networks (COM-SNETS), pages 709–713. IEEE. find
  8. Shun Moriya and Chihiro Shibata. 2018. Transfer learning method for very deep cnn for text classification and methods for its evaluation. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), volume 2, pages 153–158. IEEE. find
  9. Daša Munková, Michal Munk, and Martin Vozár. 2013. Influence of stop-words removal on sequence patterns identification within comparable corpora. In International Conference on ICT Innovations, pages 67–76. Springer. find
  10. Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020. WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. In Proceedings of the 6th Workshop on Noisy User-generated Text.
  11. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  12. Mikel Penagarikano, Amparo Varona, Luis Javier Rodriguez-Fuentes, and German Bordel. 2011. Dimensionality reduction for using high-order n-grams in svm-based phonotactic language recognition. In Twelfth Annual Conference of the International Speech Communication Association. find
  13. Georgios K Pitsilis, Heri Ramampiaro, and Helge Langseth. 2018. Detecting offensive language in tweets using deep learning. arXiv preprint arXiv:1801.04433. ↗
  14. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International workshop on natural language processing for social media, pages 1–10.
  15. Nian Shao, Yu Chen, Jin Cheng, and Wen Bin Chen. 2020. Some novel statistical time delay dynamic model by statistics data from ccdc on novel coronavirus pneumonia. Kongzhi Lilun Yu Yingyong/Control Theory and Applications, 37(4). ↗
  16. Madichetty Sreenivasulu and M Sridevi. 2020. Comparative study of statistical features to detect the target event during disaster. Big Data Mining and Analytics, 3(2):121–130. ↗
  17. Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018. Overview of the germeval 2018 shared task on the identification of offensive language. find
  18. Bo Xu, Moritz UG Kraemer, Bernardo Gutierrez, Sumiko Mekaru, Kara Sewalk, Alyssa Loskill, Lin Wang, Emily Cohn, Sarah Hill, Alexander Zarebski, et al. 2020. Open access epidemiological data from the covid-19 outbreak. The Lancet Infectious Diseases, 20(5):534. ↗
  19. Jiarui Zhang, Yingxiang Li, Juan Tian, and Tongyan Li. 2018. Lstm-cnn hybrid model for text classification. In 2018 IEEE 3rd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), pages 1675–1680. IEEE. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/s-2020-nit-covid-19-wnut-2020/