Elamur Research⌕
☰
home / papers / 2020 / CSECU-DSG at WNUT-2020 Task 2: Exploiting Ensemble of Transfer Learning and Hand-crafted Features for Identification of Informative COVID-19 English Tweets

CSECU-DSG at WNUT-2020 Task 2: Exploiting Ensemble of Transfer Learning and Hand-crafted Features for Identification of Informative COVID-19 English Tweets

Paper · 2020 · Fareen Tasneem, Jannatun Naim, Radiathun Tasnia et al.

Summary

The paper describes the University of Chittagong team's entry in WNUT-2020 Task 2. The task is to decide whether an English tweet about COVID-19 is INFORMATIVE (new cases, deaths, guidelines, vaccine news) or UNINFORMATIVE. Each tweet is represented by several feature blocks:

  • frozen sentence embeddings from BERT-Large, RoBERTa-Large and InferSent;
  • an n-gram bag-of-words vector built after heavy normalization of noisy tweet text;
  • 18 hand-crafted cues about surface form, Twitter features, sentiment and COVID topics.

Different combinations of these blocks are fed to multilayer perceptrons. One submission fuses seven such MLPs by majority vote. The other is a single MLP. Both scored about 0.82 F1 on the 10K-tweet COVID-19 Tweet dataset. That beat the organizers' FastText baseline by roughly 7 points but trailed the winner by about 9. An ablation shows RoBERTa contributes the most. The ensemble brought almost no gain, which the authors attribute to feature variants that were too similar and to missing feature selection.

Key points

  • Dataset: the WNUT-2020 COVID-19 Tweet dataset has 10K English tweets: 7K train, 1K dev and 2K test (944 informative, 1056 uninformative). F1 is the primary metric.

  • Encoders:

    • BERT-Large Uncased: 1024-d vectors.
    • RoBERTa-Large: 1024-d vectors.
    • InferSent with fastText: 4096-d vectors.
    • n-gram BoW: range 1–8, capped at 5500 features.

  • Classifier: an MLP with two hidden layers of 500 and 2000 ReLU units, alpha 0.003 and up to 1000 iterations.

  • Test results:

    SystemF1Accuracy
    CSECU-DSG#1 (7-MLP majority vote)0.81980.8290
    CSECU-DSG#2 (single MLP, setting S2)0.81560.8255
    NutCracker (winner)0.9096—
    FastText baseline0.7503—

  • Ablation: on the dev set, CSECU-DSG#2 reaches F1 0.8598. Removing a feature block costs:

    Removed blockF1 drop (points)
    RoBERTa4.53
    BERT3.61
    n-gram3.53
    other hand-crafted features1.24

Interesting ideas

  • I1. Cleaning noisy text before feature extraction pays off: dictionary-based repair of words like "2day", hashtag splitting ("#CovidPandemic" → "covid pandemic"), emoji converted to words, and digits spelled out.
  • I2. One targeted cue checks for a numeral appearing together with words like "deaths" or "cases", which crudely signals a reported quantity of affected people.
  • I3. The task separates tweets that report news from tweets that only react or comment, using small hand-built lexicons of domain verbs and sentiment cues.
  • I4. The authors report honestly that voting over seven classifiers barely beat one model, because the members differed only in minor hand-crafted features.
  • I5. The ablation removes one feature family at a time and reports the F1 drop, giving a clear ranking of what each source contributes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): a pre-tokenizer pass could split hashtags and map emoji and slang to standard words. The original span would be kept so the change stays explainable and the tagger sees ordinary English.
  • I2 (world): the world builder could flag sentences that attach a measured count to a set of entities, marking reportable states (e.g. "21 people tested positive").
  • I3 (frames): speech-act labelling could tell assertions of new facts apart from opinion or irony. Lexicons of reporting and event verbs would serve as transparent evidence for that label.
  • I4 (learn): when silver data is agreed by several witness parsers, it is worth checking that the witnesses truly disagree in useful ways. Otherwise their consensus adds confidence without adding information.
  • I5 (learn): the same leave-one-component-out protocol could measure how much each knowledge source (VerbNet roles, WordNet ontology, repair rules) actually helps answer accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ChittagongChittagong · Bangladesh4 works · 5 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Fareen Tasneem, Jannatun Naim, Radiathun Tasnia, Tashin Hossain and Abu Nowshed Chy. 2020. CSECU-DSG at WNUT-2020 Task 2: Exploiting Ensemble of Transfer Learning and Hand-crafted Features for Identification of Informative COVID-19 English Tweets. Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wnut-1.55

@inproceedings{tasneem2020csecu,
  title = {{CSECU-DSG at WNUT-2020 Task 2: Exploiting Ensemble of Transfer Learning and Hand-crafted Features for Identification of Informative COVID-19 English Tweets}},
  author = {Fareen Tasneem, Jannatun Naim, Radiathun Tasnia, Tashin Hossain and Abu Nowshed Chy},
  booktitle = {Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)},
  year = {2020},
  doi = {10.18653/v1/2020.wnut-1.55},
  url = {https://doi.org/10.18653/v1/2020.wnut-1.55}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christos Baziotis, Nikos Pelekis, and Christos Doulk-eridis. 2017. Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747–754.
  2. Cornelia Caragea, Adrian Silvescu, and Andrea H Tapia. 2016. Identifying informative messages in disaster events using convolutional neural networks. In International Conference on Information Systems for Crisis Response and Management, pages 137– 147. find
  3. Abu Nowshed Chy, Md Zia Ullah, and Masaki Aono. 2017. Microblog retrieval using ensemble of feature sets through supervised feature selection. IE-ICE TRANSACTIONS on Information and Systems, 100(4):793–806. ↗
  4. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL:HLT), pages 4171–4186. ↗
  6. Noushin Fadaei, Chanjong Im, Sandip Modha, and Thomas Mandl. 2018. Daiict-hildesheim@ information retrieval from microblogs during disasters (irmidis 2018). In Forum for Information Retrieval Evaluation (FIRE) (Working Notes), pages 15–17. find
  7. Bo Han, Paul Cook, and Timothy Baldwin. 2012. Automatically constructing a normalisation dictionary for microblogs. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 421–432. Association for Computational Linguistics (ACL). (ACL): Long Papers-Volume 1, pages 1035–1044. Association for Computational Linguistics (ACL). find
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  9. Steven Loria, P Keen, M Honnibal, R Yankovsky, D Karesh, E Dempsey, et al. 2014. Textblob: simplified text processing. Secondary TextBlob: simplified text processing, 3. find
  10. Venkata Kishore Neppalli, Cornelia Caragea, and Doina Caragea. 2018. Deep neural networks versus naive bayes classifiers for identifying informative tweets during disasters. In ISCRAM. find
  11. Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020. WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. In Proceedings of the 6th Workshop on Noisy User-generated Text.
  12. Endang Wahyu Pamungkas and Viviana Patti. 2018. # nondicevosulserio at semeval-2018 task 3: Exploiting emojis and affective content for irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 649–654.
  13. Lior Rokach. 2010. Ensemble-based classifiers. Artificial Intelligence Review, 33(1-2):1–39. ↗
  14. Umme Aymun Siddiqua, Tanveer Ahsan, and Abu Nowshed Chy. 2016. Combining a rule-based classifier with ensemble of feature sets and machine learning techniques for sentiment analysis on microblog. In 2016 19th International Conference on Computer and Information Technology (ICCIT), pages 304–309. IEEE. find
  15. Kevin Swanberg, Madiha Mirza, Ted Pedersen, and Zhenduo Wang. 2018. Alanis at semeval-2018 task 3: A feature engineering approach to irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 507–511.
  16. Alexander Hogenboom, Danella Bal, Flavius Frasincar, Malissa Bal, Franciska De Jong, and Uzay Kaymak. 2015. Exploiting emoticons in polarity classification of text. Journal of Web Engineering, 14(1-2):22–40. ↗
  17. Mike Thelwall, Kevan Buckley, and Georgios Paltoglou. 2012. Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology (JASIST), 63(1):163–173. ↗
  18. Tamanna Hossain, Robert L Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sameer Singh, and Sean Young. 2020. Detecting covid-19 misinformation on social media. In ACL 2020 Workshop on Natural Language Processing for COVID-19 (NLP-COVID). find
  19. Junpei Zhou Xinyu Wang, Po-yao Huang, and Alexander Hauptmann. 2019. Cmu-informedia at trec 2019 incident streams track. In Proceedings of the 28th Text REtrieval Conference (TREC). NIST. find
  20. Fei Liu, Fuliang Weng, and Xiao Jiang. 2012. A broad-coverage normalization system for social media language. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics Terry Windeatt. 2006. Accuracy/diversity and ensemble mlp classifier design. IEEE Transactions on Neural Networks, 17(5):1194–1211. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tasneem-2020-csecu-dsg-wnut-2020-task/