Elamur Research⌕
☰
home / papers / 2021 / Exploring Text-transformers in AAAI 2021 Shared Task: COVID-19 Fake News Detection in English

Exploring Text-transformers in AAAI 2021 Shared Task: COVID-19 Fake News Detection in English

Paper · 2021 · Xiangyang Li, Yu Xia, Xiang Long et al.

Summary

The paper describes the third-place system in the Constraint@AAAI 2021 shared task. The task was binary classification of English social-media posts about COVID-19 (from Twitter, Facebook, Instagram and similar sources) as fake or real. The authors trained on the official dataset plus extra data they gathered online. An exploratory look at the data showed that real news outnumbers fake news and that URL fragments are among the most frequent tokens.

They compared two approaches:

  • a baseline Text-RNN, a bidirectional LSTM over GloVe embeddings;
  • Text-Transformers, which fine-tunes several pretrained encoders (BERT, Ernie, Roberta, XL-net, Electra, and the domain-adapted Covid-Twitter-Bert) inside five-fold cross-validation.

Assigning a different backbone to each fold and soft-voting across all fold models worked better than reusing one architecture for every fold. Adding confidently predicted test items back into training as pseudo-labels improved results further. The final ensemble reached a weighted F1 of about 0.986 on the test set and placed 3rd of 421 teams. The work shows that disciplined ensembling and training tricks can matter as much as the choice of backbone.

Key points

  • Weighted F1 rose step by step:
    • TextRNN: 0.926
    • five-fold single-model: about 0.975
    • five-fold five-model ensemble: 0.981
    • with pseudo-labels: 0.985 (0.9859 reported in the abstract)
  • Pseudo-labelling added test items predicted with probability above 0.95. Soft voting weighted each fold model by its best validation F1.
  • Transformer training used several standard tricks:
    • label smoothing with ε = 0.01;
    • linear warm-up from 1e-6 to 5e-5 over 6 epochs, then cosine decay back to 1e-6 over 6 epochs;
    • max length 140, 12 epochs per fold.
  • Covid-Twitter-Bert supplied domain pretraining on COVID-19 tweets. The baseline used 200-dimensional GloVe vectors.
  • The confusion matrix showed slightly more false negatives than false positives, meaning fake items labelled real were the more common error.

Interesting ideas

  • I1. Instead of training the same architecture in every cross-validation fold, give each fold a different backbone and keep all fold models for the final vote; this gains both data coverage and model diversity at once.
  • I2. Self-training with a strict confidence threshold (keep only predictions above 0.95) gives cheap extra training data while limiting how much noise gets in.
  • I3. In soft voting, weight each model by its validation score rather than counting every voter equally, so stronger members have more say.
  • I4. A simple frequency count over the corpus exposed artifacts ('https', 'co' from links) that would otherwise pollute features, and this motivated removing URLs before training.
  • I5. Checking the direction of errors in the confusion matrix, not just the overall score, showed which mistake type dominates.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when building silver data from witness parsers, train each witness on a different split and with a different architecture, so their agreement is more meaningful than agreement among near-clones.
  • I2 (learn): admit unlabeled sentences into the silver treebank only when witness parsers agree with very high confidence, and grow the set in rounds.
  • I3 (tree): when combining arc and label votes from the perceptron parser and gparse, weight each parser by its held-out LAS, ideally per relation type.
  • I4 (tokens): run frequency audits on new input domains so the tokenizer recognises URLs and similar artifacts as single units before they reach tagging.
  • I5 (answer): report asymmetric error rates for the readers (for example, wrongly saying "yes" versus wrongly saying "no" on fable QA) to find systematic biases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China100 works · 35,618 citations🎓Beijing University of Posts and TelecommunicationsBeijing · China22 works · 32,364 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Information Systems · 523 works Sociology and Political Science · 221 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Xiangyang Li, Yu Xia, Xiang Long, Zheng Li and Sujian Li. 2021. Exploring Text-transformers in AAAI 2021 Shared Task: COVID-19 Fake News Detection in English. arXiv:2101.02359 · First International Workshop, CONSTRAINT 2021 co-located with AAAI 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2101.02359 · arXiv 2101.02359

@misc{li2021exploring,
  title = {{Exploring Text-transformers in AAAI 2021 Shared Task: COVID-19 Fake News Detection in English}},
  author = {Xiangyang Li, Yu Xia, Xiang Long, Zheng Li and Sujian Li},
  year = {2021},
  doi = {10.48550/arxiv.2101.02359},
  eprint = {2101.02359},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2101.02359}
}

References

3 references, 3 of them with a card of ours. ↗ leads to the work.

  1. 1. Ceron, W., de Lima-Santos, M.F., Quiles, M.G.: Fake news agenda in the era of covid-19: Identifying trends through fact-checking content. Online Social Networks and Media p. 100116 (2020) 2. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) 3. Hamid, A., Shiekh, N., Said, N., Ahmad, K., Gul, A., Hassan, L., Al-Fuqaha, A.: Fake news detection in social media using graph neural networks and nlp techniques: A covid-19 use-case (2020) 4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016) 5. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735– 1780 (1997) 6. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) ↗ our card
  2. X. Li et al. 7. Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016) 8. Mosteller, F., Tukey, J.W.: Data analysis, including statistics. Handbook of social psychology 2, 80–203 (1968) 9. Patwa, P., Sharma, S., PYKL, S., Guptha, V., Kumari, G., Akhtar, M.S., Ekbal, A., Das, A., Chakraborty, T.: Fighting an infodemic: Covid-19 fake news dataset. arXiv preprint arXiv:2011.03327 (2020) 10. Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532–1543 (2014) 11. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018) 12. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018) 13. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019) 14. Sun, C., Qiu, X., Xu, Y., Huang, X.: How to fine-tune bert for text classification? In: China National Conference on Chinese Computational Linguistics. pp. 194–206. ↗ our card
  3. Springer (2019) 15. Sun, Y., Wang, S., Li, Y.K., Feng, S., Tian, H., Wu, H., Wang, H.: Ernie 2.0: A continual pre-training framework for language understanding. In: AAAI. pp. 8968–8975 (2020) 16. Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., Wu, H.: Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223 (2019) 17. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261 (2016) 18. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017) 19. Yuan, C., Ma, Q., Zhou, W., Han, J., Hu, S.: Early detection of fake news by utilizing the credibility of news, publishers, and users based on weakly supervised learning (2020) ↗ our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2021-exploring-text-transformers-aaai-2021/