Elamur Research⌕
☰
home / papers / 2020 / IDSOU at WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets

IDSOU at WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets

Paper · 2020 · Sora Ohashi, Tomoyuki Kajiwara, Chenhui Chu et al.

Summary

The paper describes the IDSOU team's system for WNUT-2020 Task 2. The task is a binary decision: is an English COVID-19 tweet informative, meaning it reports recovered, suspected, confirmed or fatal cases or where those cases were or travelled? The organizers released 10,000 labelled tweets, and systems are ranked by F1 on the informative class.

The authors fine-tuned 16 pre-trained transformers. Most are general-domain (BERT, XLNet, RoBERTa, XLM-RoBERTa, ALBERT, BART). The rest are domain-specific (BioBERT, ClinicalBERT, COVID-Twitter-BERT). Each model was trained three times, once with each of three objectives: cross-entropy, negative supervision and Dice loss. That gives 48 classifiers. A random forest then stacks their predicted probabilities for the informative class.

On the development set, the full ensemble beat the best single model. On the official test set the system placed 16th of 55 teams, well above the fastText baseline. The study's main practical value is showing two things: pretraining on in-domain social media text matters more than pretraining on topic-related expert text, and varying the loss function is a cheap way to make ensemble members more diverse.

Key points

  • Data split: 7,000 train, 1,000 dev and 2,000 test tweets, with the two classes roughly balanced (e.g. 3,303 informative vs 3,697 uninformative in train). Precision, recall and F1 are computed only for the informative class.
  • Best single models on dev: COVID-Twitter-BERT was strongest, reaching F1 0.922 with cross-entropy and 0.926 with Dice loss. BioBERT and ClinicalBERT did not beat the general-domain models.
  • Ensemble gains on dev: BERT-large alone scored 0.898. An ensemble of 16 cross-entropy models scored 0.929, and all 48 models (XE+NS+DS) scored 0.933.
  • Loss details: Negative supervision adds a cosine-similarity penalty against M=2 examples with a different label; it improved 6 of the 16 models. Dice loss uses smoothing γ=1.
  • Stacker and test result: The random forest was tuned with 5-fold cross-validation on the dev set. Official test results were F1 0.8964, precision 0.8988 and recall 0.8941. The winning team scored 0.9096 and the baseline 0.7503.

Interesting ideas

  • I1. Using different training objectives on the same model is a cheap source of ensemble diversity, since it adds members without needing new architectures or data.
  • I2. A small learned meta-classifier over each member's confidence replaces naive voting and can learn which member to trust in which situation.
  • I3. The domain that helped was the register of the text (lay tweets), not its topic (medicine); the biomedical encoders gave no gain because the authors are mostly non-experts.
  • I4. Dice loss narrows the gap between what is optimized during training and the F1 metric used for ranking, which matters when one class is the focus.
  • I5. Negative supervision explicitly pushes apart the representations of examples with different labels, making classes separate more cleanly in embedding space.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): several gparse checkpoints trained with different losses could act as extra, partly independent witness parsers when building agreed silver trees.
  • I2 (learn): a tiny explainable arbiter (e.g. a shallow tree) over witness parsers' arc confidences could decide silver labels, with its splits readable as rules about which parser is reliable when.
  • I3 (tree): when choosing treebanks or pretraining text for parsing informal input, match genre and register (web and social UD data) rather than subject matter.
  • I4 (frames): detectors for rare phenomena such as irony or particular speech acts could be trained with an F1-aligned loss so that recall on the rare class isn't sacrificed.
  • I5 (frames): contrasting literal and ironic readings of near-identical sentences could sharpen the cues the irony or implicature module relies on.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of OsakaOsaka · Japan15 works · 517 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Sora Ohashi, Tomoyuki Kajiwara, Chenhui Chu, Noriko Takemura, Yuta Nakashima and Hajime Nagahara. 2020. IDSOU at WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wnut-1.62

@inproceedings{ohashi2020idsou,
  title = {{IDSOU at WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets}},
  author = {Sora Ohashi, Tomoyuki Kajiwara, Chenhui Chu, Noriko Takemura, Yuta Nakashima and Hajime Nagahara},
  booktitle = {Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)},
  year = {2020},
  doi = {10.18653/v1/2020.wnut-1.62},
  url = {https://doi.org/10.18653/v1/2020.wnut-1.62}
}

References

23 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly Available Clinical BERT Embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78. ↗
  2. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A Pretrained Language Model for Scientific Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3615–3620. find
  3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  4. Leo Breiman. 2001. Random Forests. Machine Learning, 45:5–32. find
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186.
  7. Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. MIMIC-III, a Freely Accessible Critical Care Database. Scientific Data, 3(160035).
  8. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations. ↗
  9. Rabindra Lamsal. 2020. Tweets Dataset. Coronavirus (COVID-19) find
  10. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In Proceedings of the Eighth International Conference on Learning Representations.
  11. Jieh-Sheng Lee and Jieh Hsiang. 2019. PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model. arXiv:1906.02124. ↗
  12. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: A Pre-trained Biomedical Language Representation Model for Biomedical Text Mining. Bioinformatics, pages 1234–1240. ↗
  13. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880.
  14. Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020. Dice Loss for Data-imbalanced NLP Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 465–476.
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692. ↗
  16. Martin Müller, Marcel Salathé, and Per E Kummervold. 2020. COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter. arXiv:2005.07503. ↗
  17. Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020. WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. In Proceedings of the 6th Workshop on Noisy User-generated Text.
  18. Sora Ohashi, Junya Takayama, Tomoyuki Kajiwara, Chenhui Chu, and Yuki Arase. 2020. Text Classification with Negative Supervision. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 351–357. find
  19. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12(85):2825–2830. ↗
  20. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 7th International Conference on Learning Representations. ↗
  21. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv:1910.03771. ↗
  22. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Proceedings of the 33rd Conference on Neural Information Processing Systems, pages 5753–5763. ↗
  23. Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books. arXiv:1506.06724. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ohashi-2020-idsou-wnut-2020-task-2/