Elamur Research⌕
☰
home / papers / 2017 / A simple but tough-to-beat baseline for the Fake News Challenge stance detection task

A simple but tough-to-beat baseline for the Fake News Challenge stance detection task

Paper · 2017 · Benjamin Riedel, Isabelle Augenstein, Georgios P. Spithourakis et al.

Summary

The paper addresses stance detection for Stage 1 of the Fake News Challenge (FNC-1). Given a headline and an article body, a system must decide whether the body agrees with, disagrees with, discusses, or is unrelated to the headline. Such systems are meant to support human fact checkers. The UCL Machine Reading (UCLMR) team built a deliberately minimal, single end-to-end model. It feeds bag-of-words term-frequency vectors of the headline and body, plus a TF-IDF cosine similarity score, into a multi-layer perceptron with one hidden layer, implemented in TensorFlow and scikit-learn. This model placed third among 50 teams. Its score was within a fraction of a point of the two winners, both of which used complex ensembles. However, the error analysis shows that most of its success comes from separating related from unrelated pairs. It handles the "agree" and especially "disagree" classes poorly, and these are the classes most useful for fact checking. The authors release their code and offer the system as a strong, transparent reference baseline for FNC-1.

Key points

  • Features: TF vectors of the headline and body use a 5,000-word vocabulary with stop words removed but negation terms kept. Together with the TF-IDF cosine similarity, they form a 10,001-dimensional input.
  • Model and training: The MLP has one hidden layer of 100 ReLU units and a softmax output. Training uses cross-entropy loss, L2 regularisation of 0.0001, dropout with keep probability 0.6, the Adam optimiser with a learning rate of 0.01, gradient clipping at 5, batches of 500, and 90 epochs.
  • Leaderboard: The FNC-1 score was 81.72%, placing third. The winners were "SOLAT in the SWEN" (Talos Intelligence) at 82.02% and "Athene" (TU Darmstadt) at 81.97%. The official baseline scored 75.20%.
  • Per-class accuracy: Related vs. unrelated accuracy was 96.55%. Class accuracy was 97.90% for "unrelated", 81.38% for "discuss", 44.04% for "agree", and only 6.60% for "disagree".
  • Code and origins: The code is available at github.com/uclmr/fakenewschallenge. The setup builds on earlier FNC-1 work by Davis and Proctor at Stanford.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 1,035 works

Cite

Cite as Benjamin Riedel, Isabelle Augenstein, Georgios P. Spithourakis and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the Fake News Challenge stance detection task. arXiv:1707.03264.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1707.03264 · arXiv 1707.03264

@misc{riedel2017simple,
  title = {{A simple but tough-to-beat baseline for the Fake News Challenge stance detection task}},
  author = {Benjamin Riedel, Isabelle Augenstein, Georgios P. Spithourakis and Sebastian Riedel},
  year = {2017},
  doi = {10.48550/arxiv.1707.03264},
  eprint = {1707.03264},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1707.03264}
}

References

16 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B.… find
  2. A. Agrawal, D. Chin, and K. Chen. Cosine siamese models for stance detection, 2017. URL http://web.stanford.edu/class/cs224n/reports/2759862.pdf. ↗
  3. S. Arora, Y. Liang, and T. Ma. A simple but tough-to-beat baseline for sentence embeddings. In International Conference for Learning Representations, 2017. find
  4. S. Baird, D. Sibley, and Y. Pan. Talos targets disinformation with Fake News Challenge victory, 2017. URL https://blog.talosintelligence.com/2017/06/talos-fake-news-challenge.html. ↗
  5. G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 1989. find
  6. R. Davis and C. Proctor. Fake news, real consequences: Recruiting neural networks for the fight against fake news, 2017. URL https://web.stanford.edu/class/cs224n/reports/2761239.pdf. ↗
  7. B. Galbraith, H. Iqbal, H. van Veen, D. Rao, J. Thorne, and Y. Pan. A baseline implementation for FNC-1, 2017. URL https://github.com/FakeNewsChallenge/fnc-1-baseline. ↗
  8. R. H. R. Hahnloser, R. Sarpeshkar, M. A. Mahowald, R. J. Douglas, and H. S. Seung. Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature, 2000. find
  9. A. Hanselowski, A. PVS, B. Schiller, and F. Caspelherr. Description of the system developed by team Athene in the FNC-1, 2017. URL https://github.com/hanselowski/athene_system/blob/master/system_description_athene.pdf. ↗
  10. K. S. Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 1972. find
  11. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2015.
  12. H. P. Luhn. A statistical approach to mechanized encoding and searching of literary information. IBM Journal of Research and Development, 1957. find
  13. R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, 2013. find
  14. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, and M. P. E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011. find
  15. D. Pomerleau and D. Rao. fakenewschallenge.org/. Fake News Challenge, 2017. find
  16. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014. URL http://www.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/riedel-2017-simple-but-tough-beat-baseline/