Elamur Research⌕
☰
home / papers / 2019 / GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification

GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification

Summary

Fact verification (FV) means judging a claim as SUPPORTED, REFUTED or NOT ENOUGH INFO using sentences retrieved from Wikipedia. Many claims can only be resolved by combining several sentences. Earlier FEVER systems treated the task as NLI but either concatenated all evidence or scored each claim–evidence pair separately, so the evidence pieces never exchanged information. The authors propose GEAR, which builds on a three-stage pipeline: document retrieval, sentence selection and claim verification. The first two stages reuse the approach of Hanselowski et al. (2018), with an added relevance-score threshold that removes noisy sentences. For verification, each evidence–claim pair is encoded with BERT. The encoded sentences become nodes in a fully connected graph, and an evidence reasoning network (ERNet) passes messages between them. An aggregator then pools the node states for classification. On the FEVER benchmark, GEAR reaches a test FEVER score of 67.10% and beats a strong BERT concatenation baseline. This indicates that explicit reasoning across evidence helps multi-sentence verification.

Key points

  • Three aggregators are proposed: attention (guided by the claim representation), max and mean. The best final model uses one ERNet layer with the attention aggregator and a sentence filter threshold of 10⁻³.
  • Retrieval follows the UKP-Athene method. Entities found by an AllenNLP constituency parser are used as MediaWiki API queries, keeping the top 7 results per query. An ESIM-based sentence ranker, ensembled over 10 seeds, then keeps at most 5 sentences.
  • The full pipeline scores 67.10% FEVER on the test set, 1.46% above BERT-Concat. BERT fine-tuning systems already exceed earlier shared-task entries by roughly 1%.
  • An evidence-enhanced dev set, with ground-truth evidence injected, raises label accuracy by more than 1.4%. This shows that upstream retrieval remains a bottleneck.
  • Code is released at github.com/thunlp/GEAR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

94 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1085

@inproceedings{zhou2019gear,
  title = {{GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification}},
  author = {Zhou, Jie and Han, Xu and Yang, Cheng and Liu, Zhiyuan and Wang, Lifeng and Li, Changcheng and Sun, Maosong},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1085},
  url = {https://doi.org/10.18653/v1/p19-1085}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gabor Angeli and Christopher D Manning. 2014. Naturalli: Natural logic inference for common sense reasoning. In Proceedings of EMNLP, pages 534– 545. find
  2. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  3. Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of ICLR. find our card
  4. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP, pages 632–642. find
  5. Jackson Luken, Nanjiang Jiang, and Marie-Catherine de Marneffe. 2018. Qed: A fact verification system for the fever shared task. In Proceedings of the First Workshop on FEVER, pages 156–160. find
  6. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017a. Reading Wikipedia to answer open-domain questions. In Proceedings of ACL, pages 1870–1879. find our card
  7. Christopher Malon. 2018. Team papelo: Transformer networks at fever. In Proceedings of the First Workshop on FEVER, pages 109–113. find
  8. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017b. Enhanced lstm for Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT, pages 1112–1122. find
  9. Tsendsuren Munkhdalai and Hong Yu. 2017. Neural tree indexers for text understanding. In Proceedings of EACL, pages 11–21. find
  10. Yixin Nie and Mohit Bansal. 2017. Shortcut-stacked sentence encoders for multi-domain inference. In Proceedings of ACL, Workshop on Evaluating Vector Space Representations for NLP, pages 41–45. find
  11. Wenpeng Yin and Dan Roth. 2018. Twowingos: A two-wing optimization strategy for evidential claim verification. In Proceedings of EMNLP, pages 105– 114. find our card
  12. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of AAAI. find
  13. Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Ucl machine reading group: Four factor framework for fact finding (hexaf). In Proceedings of the First Workshop on FEVER, pages 97–102. find
  14. Rasmus Palm, Ulrich Paquet, and Ole Winther. 2018. Recurrent relational networks. In Proceedings of NIPS, pages 3368–3378. find
  15. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of EMNLP, pages 2249–2255. find our card
  16. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237. find
  17. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. In Proceedings of Technical report, OpenAI. find
  18. Lei Sha, Baobao Chang, Zhifang Sui, and Sujian Li. 2016. Reading and thinking: Re-read lstm unit for textual entailment recognition. In Proceedings of COLING, pages 2870–2879. find
  19. Yi Tay, Anh Tuan Luu, and Siu Cheung Hui. 2018. Compare, compress and propagate: Enhancing neural architectures with alignment factorization for natural language inference. In Proceedings of EMNLP, pages 1565–1575. find
  20. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of NAACL-HLT, pages 809–819. find our card
  21. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b. The Fact Extraction and VERification (FEVER) shared task. In Proceedings of the First Workshop on FEVER, pages 1–9. find our card
  22. Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In Proceedings of ICLR. find
  23. Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE TKDE, 29(12):2724–2743. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2019-gear-graph-based-evidence-aggregating/