Elamur Research⌕
☰
home / papers / 2020 / Fine-grained Fact Verification with Kernel Graph Attention Network

Fine-grained Fact Verification with Kernel Graph Attention Network

Summary

Automatic fact checking must decide whether a claim is supported, refuted, or lacks sufficient information, using sentences retrieved from Wikipedia. This is hard for two reasons. The retrieved evidence is noisy, and false claims are often plausible statements that differ from the truth only in subtle details.

The authors propose Kernel Graph Attention Network (KGAT). It treats every claim–evidence pair as a node in a fully connected graph and initialises each node with BERT. It then replaces the usual dot-product attention with Gaussian matching kernels in two places:

  • Edge kernels choose which tokens from neighbouring nodes are passed along during multi-layer propagation.
  • Node kernels weight how much each evidence node contributes to the final label.

On the FEVER benchmark, KGAT reaches a 70.38% FEVER score, ahead of earlier BERT- and GNN-based systems such as GEAR. The analyses show that kernel attention is much more concentrated than dot-product attention. This helps the model filter out noisy evidence and improves reasoning over several sentences. It also makes the model's decisions easier to interpret.

Key points

  • Task and data: FEVER, with three labels (SUPPORTS, REFUTES, NOT ENOUGH INFO) and Wikipedia as the evidence source. KGAT targets the claim verification step of the usual pipeline (document retrieval → sentence retrieval → verification).
  • Method: The claim label probability is a sum over nodes. Each node's label prediction is weighted by its evidence-selection probability. The kernels operate on a cosine-similarity matrix built from BERT token representations.
  • Main result: 70.38% FEVER score, with the clearest gains on claims that need several evidence pieces.
  • Evidence selection: The sentence ranked first by node kernels covers nearly 80% of ground-truth evidence. GAT's node attention, by contrast, is close to uniform.
  • Code: Released at github.com/thunlp/KernelGAT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

99 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 93 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.655

@inproceedings{liu2020fine,
  title = {{Fine-grained Fact Verification with Kernel Graph Attention Network}},
  author = {Liu, Zhenghao and Xiong, Chenyan and Sun, Maosong and Liu, Zhiyuan},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.655},
  url = {https://doi.org/10.18653/v1/2020.acl-main.655}
}

References

37 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017a. Reading wikipedia to answer open-domain questions. In Proceedings of ACL, pages 1870–1879. find our card
  2. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017b. Enhanced LSTM for natural language inference. In Proceedings of ACL, pages 1657–1668. find
  3. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286. find
  4. Zhuyun Dai and Jamie Callan. 2019. Deeper text understanding for ir with contextual neural language modeling. In Proceedings of SIGIR, pages 985–988. find
  5. Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional neural networks for soft-matching n-grams in ad-hoc search. In Proceedings of WSDM, pages 126–134. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171– 4186. find
  7. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6. find our card
  8. Reza Ghaeini, Sadid A Hasan, Vivek Datla, Joey Liu, Kathy Lee, Ashequl Qadir, Yuan Ling, Aaditya Prakash, Xiaoli Fern, and Oladimeji Farri. 2018. Dr-bilstm: Dependent reading bidirectional LSTM for natural language inference. In Proceedings of NAACL, pages 1460–1469. find
  9. Jiafeng Guo, Yixing Fan, Qingyao Ai, and W.Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of CIKM, pages 55– 64. find
  10. Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. UKP-athene: Multi-sentence textual entailment for claim verification. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 103–108. find our card
  11. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Proceedings of NIPS, pages 2042–2050. find
  12. Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of ICLR. find our card
  13. Boris Knyazev, Graham W Taylor, and Mohamed R Amer. 2019. Understanding attention and generalization in graph neural networks. In Proceedings of NeurIPS, pages 4202–4212. find
  14. Tianda Li, Xiaodan Zhu, Quan Liu, Qian Chen, Zhigang Chen, and Si Wei. 2019. Several experiments on investigating pretraining and knowledge-enhanced models for natural language inference. arXiv preprint arXiv:1904.12104. — [arXiv:1904.12104](https://arxiv.org/abs/1904.12104) find
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  16. Jackson Luken, Nanjiang Jiang, and Marie-Catherine de Marneffe. 2018. QED: A fact verification system for the fever shared task. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 156–160. find
  17. Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019. CEDR: contextualized embeddings for document ranking. In Proceedings of SIGIR, pages 1101–1104. find
  18. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019a. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of AAAI, pages 6859–6866. find
  19. Yixin Nie, Songhe Wang, and Mohit Bansal. 2019b. Revealing the importance of semantic retrieval for machine reading at scale. In Proceedings of EMNLP, pages 2553–2566. find
  20. Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In Proceedings of AAAI, pages 2793–2799. find
  21. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of EMNLP, pages 2249–2255. find our card
  22. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL, pages 2227– 2237. find
  23. Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019. Understanding the behaviors of bert in ranking. arXiv preprint arXiv:1904.07531. — [arXiv:1904.07531](https://arxiv.org/abs/1904.07531) find
  24. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b. The fact extraction and verification (FEVER) shared task. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 1–9. find our card
  25. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903. find our card
  26. Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of SIGIR, pages 55–64. find
  27. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Proceedings of NeurIPS, pages 5754–5764. find our card
  28. Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential reasoning learning for language representation. arXiv preprint arXiv:2004.06870. find
  29. Wenpeng Yin and Dan Roth. 2018. TwoWingOS: A two-wing optimization strategy for evidential claim verification. In Proceedings of EMNLP, pages 105– 114. find our card
  30. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. In Proceedings of Technical report, OpenAI. find
  31. Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. UCL machine reading group: Four factor framework for fact finding (HexaF). In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 97–102. find
  32. Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE Transactions on Neural Networks, pages 61–80. find
  33. Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary. 2020. Transformer-xh: Multi-evidence reasoning with extra hop attention. In Proceedings of ICLR. find
  34. Amir Soleimani, Christof Monz, and Marcel Worring. 2019. BERT for evidence retrieval and claim verification. arXiv preprint arXiv:1910.02655. — [arXiv:1910.02655](https://arxiv.org/abs/1910.02655) find
  35. Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2019. Reasoning over semantic-level graph for fact checking. arXiv preprint arXiv:1909.03745. find our card
  36. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of NAACL, pages 809–819. find our card
  37. Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proceedings of ACL, pages 892–901. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2020-fine-grained-fact-verification-kernel/