Elamur Research⌕
☰
home / papers / 2020 / TATL at WNUT-2020 Task 2: A Transformer-based Baseline System for Identification of Informative COVID-19 English Tweets

TATL at WNUT-2020 Task 2: A Transformer-based Baseline System for Identification of Informative COVID-19 English Tweets

Paper · 2020 · Tuan Nguyen, Anh

Summary

This short paper describes the TATL system for W-NUT 2020 Shared Task 2. The task is to decide whether a COVID-19 English Tweet is informative, meaning it reports suspected, confirmed, recovered or death cases, or where and how a case happened. The author treats this as binary classification over a dataset of about 10K Tweets. Each Tweet is encoded with a pretrained Transformer. The [CLS] vector is passed through a single linear layer with a sigmoid to give the probability that the Tweet is informative. Several encoders were fine-tuned and compared: RoBERTa, XLM-RoBERTa, BERTweet and ELECTRA, in base and large sizes where available. Their outputs were then combined in two ways, unweighted probability averaging and majority voting. RoBERTa-large was the strongest single model on validation. Both ensembles beat every individual model. The submitted system reached about 0.90 F1 on the hidden test set and placed 8th among roughly 55 teams. The paper's main value is as a strong, easy-to-reproduce reference point: fine-tuning off-the-shelf models and ensembling them, with no task-specific engineering, already ranks near the top.

Key points

  • Data splits (informative / uninformative):
    • training: 3303 / 3697
    • validation: 472 / 528
    • test: 944 / 1056
  • Fine-tuning setup: transformers library, AdamW, batch size 32, learning rates from {1e-5, 2e-5, 5e-5}, 30 epochs. The best checkpoint was chosen on the validation set.
  • Validation F1 of single models:
    • XLM-RoBERTa base 0.905, large 0.906
    • RoBERTa base 0.911, large 0.918 (best single model)
    • BERTweet 0.909
    • ELECTRA base 0.907, large 0.914
  • Ensembles: on validation, averaging scored 0.927 and voting 0.922. On the test set the order flipped, with voting at 0.9008 and averaging at 0.8988.
  • Rank: 8th place. The abstract counts 55 teams, the conclusion 56.

Interesting ideas

  • I1. Combining several differently pretrained encoders through plain probability averaging or majority vote reliably beat the best single model, at no cost beyond extra inference.
  • I2. Which ensemble scheme won depended on the split: averaging led on validation, voting on the hidden test. Small dev differences should therefore not be trusted to pick a method.
  • I3. The Tweet-specific BERTweet did not beat general RoBERTa-base, and multilingual XLM-RoBERTa trailed English-only models. Domain-matched or broader pretraining should be tested, not assumed to help.
  • I4. The baseline is deliberately minimal: one pooled vector, one linear layer, standard fine-tuning. This gives a clear floor that more elaborate systems must beat to justify their complexity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Witness parsers that agree on silver data could be compared under both averaging of arc scores and hard voting, to see which consensus rule yields cleaner silver trees.
  • I2 (learn): When choosing between repair rules or training variants, report results on a held-out split separate from the one used for selection, and treat sub-point gaps as ties.
  • I3 (tree): Before adopting a domain-specific or multilingual encoder for gparse, run a direct comparison against a general English encoder on our own treebanks.
  • I4 (answer): A simple pooled-encoder classifier could serve as a negative-control baseline for school-test and fable QA, showing how much the explainable tree→frames→world pipeline actually adds.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NvidiaSanta Clara · United States25 works · 5,058 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Tuan Nguyen, Anh. 2020. TATL at WNUT-2020 Task 2: A Transformer-based Baseline System for Identification of Informative COVID-19 English Tweets. Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wnut-1.42

@inproceedings{anh2020tatl,
  title = {{TATL at WNUT-2020 Task 2: A Transformer-based Baseline System for Identification of Informative COVID-19 English Tweets}},
  author = {Tuan Nguyen, Anh},
  booktitle = {Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)},
  year = {2020},
  doi = {10.18653/v1/2020.wnut-1.42},
  url = {https://doi.org/10.18653/v1/2020.wnut-1.42}
}

References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: Pretrained language model for scientific text. In EMNLP. find
  2. Qian Chen, Zhu Zhuo, and W. Wang. 2019. Bert for joint intent classification and slot filling. ArXiv, abs/1902.10909. ↗
  3. K. Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. ArXiv, abs/2003.10555. ↗
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171– 4186.
  6. Lifu Huang, Heng Ji, Kyunghyun Cho, Ido Dagan, Sebastian Riedel, and Clare Voss. 2018. Zero-shot transfer learning for event extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2160–2170, Melbourne, Australia. Association for Computational Linguistics.
  7. Tuan Lai, Trung Bui, Nedim Lipka, and Sheng Li. 2018. Supervised transfer learning for product information question answering. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 1109–1114. IEEE. find
  8. Tuan Lai, Quan Hung Tran, Trung Bui, and Daisuke Kihara. 2019. A gated self-attention memory network for answer selection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5953–5959, Hong Kong, China. Association for Computational Linguistics. ↗
  9. Tuan Manh Lai, Quan Hung Tran, Trung Bui, and Daisuke Kihara. 2020. A simple but effective bert model for dialog state tracking on resource-limited systems. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8034–8038. IEEE. ↗
  10. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS). find
  11. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. ↗
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint, arXiv:1907.11692. ↗
  13. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In Proceedings of ICLR. find
  14. Sugeerth Murugesan, Sana Malik, Fan Du, Eunyee Koh, and Tuan Manh Lai. 2019. Deepcompare: Visual and interactive comparison of deep learning model performance. IEEE computer graphics and applications, 39(5):47–59. ↗
  15. Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. Findings of EMNLP.
  16. Dat Quoc Nguyen, Thanh Vu, and A. Nguyen. 2020a. Bertweet: A pre-trained language model for english tweets. EMNLP 2020.
  17. Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020b. WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. In Proceedings of the 6th Workshop on Noisy User-generated Text.
  18. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  19. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. ↗
  20. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv preprint, arXiv:1910.03771. ↗
  21. Qizhe Xie, E. Hovy, Minh-Thang Luong, and Quoc V. Le. 2020. Self-training with noisy student improves imagenet classification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695. find
  22. Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2019. A compare-aggregate model with latent clustering for answer selection. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 2093–2096. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2020-tatl-wnut-2020-task-2/