Elamur Research⌕
☰
home / papers / 2020 / Reasoning Over Semantic-Level Graph for Fact Checking

Reasoning Over Semantic-Level Graph for Fact Checking

Summary

Automated fact checking means judging whether a textual claim is supported or contradicted by evidence, and that evidence is often spread across several sentences. Earlier systems either glued evidence sentences into one string or combined features of each sentence separately. Both approaches miss the semantic links between facts in different sentences. The authors instead build a graph from the evidence using semantic role labeling (SRL). Its nodes are verbs, arguments, and location and time phrases. Edges connect elements within one SRL tuple and also link textually overlapping phrases across tuples. Two mechanisms inject this structure into XLNet. First, word distances are recomputed from graph connectivity, so semantically linked words count as close. Second, a graph convolutional network spreads information between nodes and a graph attention network pools it for the final decision. On FEVER, both mechanisms improved results, and the full system ranked first on the public leaderboard for label accuracy and FEVER score when the paper was submitted. The work shows that explicit semantic structure adds value on top of strong pre-trained encoders. (The excerpt provided omits the exact scores.)

Key points

  • Dataset and task: FEVER has 185,445 claims checked against Wikipedia. Each claim is labelled SUPPORTED, REFUTED or NOT ENOUGH INFO. Systems are evaluated by three-way accuracy and by FEVER score, which also requires retrieving the correct evidence.

  • Pipeline: document retrieval, then selection of the top-k evidence sentences, then claim verification. The graph-based verifier is the main contribution.

  • Graph construction: sentences are parsed with the AllenNLP SRL tool, a re-implementation of the BERT-based model of Shi and Lin (2019). Nodes in different tuples are linked if one of three conditions holds:

    • they are identical;
    • one contains the other;
    • their word overlap is more than half the length of the shorter phrase.

    The claim is turned into a graph with the same procedure.

  • Model: XLNet is the backbone, with graph-based relative distances, followed by a GCN for propagation and a GAT for aggregation. Ablation experiments confirm that each graph component helps.

  • Error source and future work: irrelevant retrieved evidence can mislead the verifier. One example is a film titled "The Gifted" being confused with a TV series of the same name. Joint training of evidence selection and verification is proposed as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Semantic role labeling

Cited by

73 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.549

@inproceedings{zhong2020reasoning,
  title = {{Reasoning Over Semantic-Level Graph for Fact Checking}},
  author = {Zhong, Wanjun and Xu, Jingjing and Tang, Duyu and Xu, Zenan and Duan, Nan and Zhou, Ming and Wang, Jiahai and Yin, Jian},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.549},
  url = {https://doi.org/10.18653/v1/2020.acl-main.549}
}

References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gabor Angeli and Christopher D Manning. 2014. Naturalli: Natural logic inference for common sense reasoning. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 534–545. find
  2. Michele Banko, Michael J Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Ijcai, volume 7, pages 2670–2676. find our card
  3. Xavier Carreras and Lluı́s Màrquez. 2004. Introduction to the conll-2004 shared task: Semantic role labeling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pages 89–97. find
  4. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced lstm for natural language inference. arXiv preprint arXiv:1609.06038. find
  5. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1–220. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  7. Robert Faris, Hal Roberts, Bruce Etling, Nikki Bourassa, Ethan Zuckerman, and Yochai Benkler. 2017. Partisanship, propaganda, and disinformation: Online media and the 2016 us presidential election. find
  8. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1163–1168. find our card
  9. Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 166–175. ACM. find
  10. Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. Ukp-athene: Multi-sentence textual entailment for claim verification. arXiv preprint arXiv:1809.01479. find our card
  11. Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907. find our card
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  13. Zhenghao Liu, Chenyan Xiong, and Maosong Sun. 2019b. Kernel graph attention network for fact verification. arXiv preprint arXiv:1910.09796. — [arXiv:1910.09796](https://arxiv.org/abs/1910.09796) find
  14. Ndapandula Nakashole and Tom M Mitchell. 2014. Language-aware truth assessment of fact candidates. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1009–1019. find
  15. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6859–6866. find
  16. Ankur Padia, Francis Ferraro, and Tim Finin. 2018. Team UMBC-FEVER : Claim verification using semantic lexical resources. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 161–165, Brussels, Belgium. Association for Computational Linguistics. find
  17. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  18. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931–2937. find our card
  19. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155. find our card
  20. Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018. Improved semantic-aware network embedding with fine-grained word alignment. arXiv preprint arXiv:1808.09633. find
  21. Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255. find our card
  22. Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence. find
  23. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355. find our card
  24. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b. The fact extraction and verification (fever) shared task. arXiv preprint arXiv:1811.10971. find our card
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  26. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903. find our card
  27. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhong-2020-reasoning-over-semantic-level-graph/