Elamur Research⌕
☰
home / papers / 2018 / The Fact Extraction and VERification (FEVER) Shared Task

The Fact Extraction and VERification (FEVER) Shared Task

Summary

The FEVER shared task targets automatic claim verification, which complements information extraction: instead of only pulling facts out of text, a system must check whether a statement holds. Given a short human-written claim, participants had to find justifying sentences in Wikipedia and decide among three labels: SUPPORTED, REFUTED, or NOT ENOUGH INFO. The task differs from classic NLI because evidence must be retrieved from a large corpus rather than being supplied. Systems were scored with the FEVER score, which counts a label as correct only when it comes with at least one complete evidence set. Evaluation used a blind, class-balanced test set hosted on Codalab. Twenty-three teams took part, and 19 of them beat the published baseline. The winner, UNC-NLP, reached 64.21%. Most systems used a three-stage pipeline of document retrieval, sentence selection and inference. After the competition, the organizers annotated evidence returned by systems to fill gaps in the original gold evidence. This matters because human annotators had missed valid evidence, so the benchmark itself improves through the task.

Key points

  • The dataset has 185,445 claims checked against the introductory sections of Wikipedia pages. In 16.82% of cases the evidence requires combining several sentences. Dev and test sets each contain 6,666 claims per class.
  • The competition received 86 submissions from 23 teams over a four-day window, with at most 10 submissions per team. UNC-NLP won with a 64.21% FEVER score and 68.21% label accuracy. The baseline scored 27.45%.
  • Papelo achieved the best evidence precision (92.18%) and F1 (64.85%). Athene UKP TU Darmstadt had the best recall among described systems (85.19%), followed by UCL Machine Reading Group (82.84%).
  • Common techniques:
    • Document selection: entity and noun-phrase extraction, or capitalized expressions, sent to the Wikipedia/Google search APIs or Lucene/Solr.
    • Sentence selection: ESIM-style or Decomposable Attention classifiers.
    • Inference: supervised NLI models, sometimes enriched with ELMo or WordNet features.
  • Post-hoc annotation produced 1,003 judgements for 618 claims. It found 87 NOT ENOUGH INFO claims that should be relabeled and 308 new evidence sets.

Interesting ideas

  • I1. Evidence-conditioned metric: credit for a correct label is given only when it is backed by a complete evidence set. This stops systems from scoring well by guessing labels without grounding. It could transfer to QA or legal and medical classification, where justification matters.
  • I2. Using system outputs to repair gold data: evidence retrieved by participants but absent from the gold set was sent to annotators. Claims were sampled more often when many systems got them wrong, and evidence was sampled more often from stronger systems. This is a cheap way to raise recall in any benchmark whose annotations are known to be incomplete.
  • I3. Page popularity as a ranking signal: UNC-NLP ranked candidate Wikipedia pages using viewership statistics. This simple prior is useful for entity disambiguation in any Wikipedia-based retrieval.
  • I4. Splitting data by source page: train, dev and test were separated by the Wikipedia page each claim was generated from. This prevents leakage between near-duplicate claims and is a good practice for any dataset built from paraphrased source texts.
  • I5. Ways to combine multiple evidence sentences: teams either concatenated the evidence, classified each claim–sentence pair and then aggregated, or pooled per-pair encodings. Comparing these options is a useful design checklist for multi-document reasoning tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

114 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 104 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-5501

@inproceedings{thorne2018fact,
  title = {{The Fact Extraction and VERification (FEVER) Shared Task}},
  author = {Thorne, James and Vlachos, Andreas and Cocarascu, Oana and Christodoulopoulos, Christos and Mittal, Arpit},
  booktitle = {Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)},
  year = {2018},
  doi = {10.18653/v1/w18-5501},
  url = {https://doi.org/10.18653/v1/w18-5501}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough to beat baseline for sentence embeddings. In Iclr, pages 1–14. find
  2. Sean Baird, Doug Sibley, and Yuxi Pan. 2017. Cisco’s Talos Intelligence Group Blog Talos Targets Disinformation with Fake News Challenge Victory. find
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
  4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. Proc. of ACL’17. find our card
  5. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced LSTM for Natural Language Inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1657–1668, Vancouver, Canada. find
  6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen,… find our card
  7. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering, 15(4):i–xvii. find
  8. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SCITAIL: A Textual Entailment Dataset from Science Question Answering. Aaai. find
  9. Matt J Kusner, Yu Sun, Nicholas I Kolkin, and Kilian Q Weinberger. 2015. From Word Embeddings To Document Distances. Proceedings of The 32nd International Conference on Machine Learning, 37:957– 966. find
  10. George a. Miller. 1995. WordNet: a lexical database for English. Communications of the ACM, 38(11):39–41. find our card
  11. Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference. pages 2249–2255. find our card
  12. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237, New Orleans, Louisiana. find
  13. Alec Radford and Tim Salimans. 2018. Improving Language Understanding by Generative Pre-Training. arXiv, pages 1–12. find
  14. Gerard Salton, Edward A. Fox, and Harry Wu. 1983. Extended boolean information retrieval. Commun. ACM, 26(11):1022–1036. find
  15. Vladimir Svetnik, Andy Liaw, Christopher Tong, J Christopher Culberson, Robert P Sheridan, and Bradley P Feuston. 2003. Random forest: a classification and regression tool for compound classification and qsar modeling. Journal of chemical information and computer sciences, 43(6):1947–1958. find
  16. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT. find our card
  17. Yi Yang and Ming-Wei Chang. 2015. S-MART: Novel Tree-based Structured Learning Algorithms Applied to Tweet Entity Linking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 504–5… find
  18. Wenpeng Yin, Hinrich Schütze, and Dan Roth. 2018. End-Task Oriented Textual Entailment via Deep Explorations of Inter-Sentence Interactions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 540–545, Melbourne, Australia. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thorne-2018-fact-extraction-verification-fever-shared/