Elamur Research⌕
☰
home / papers / 2018 / TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification

TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification

Summary

The paper tackles evidential claim verification: given a claim and a large text collection, a system must find the sentences that support or refute the claim and then decide whether the claim is supported, refuted, or lacks enough information. Earlier systems treat this as a pipeline. They first retrieve a fixed number of similar sentences and then run textual entailment on them. Yin and Roth argue that the two steps depend on each other, so they propose TwoWingOS. This framework maps a shared claim representation into two output spaces at once: a binary selection vector over candidate sentences and a one-hot verdict. Each candidate is scored independently, using CNN encodings and claim-aware attentive convolution, so the evidence set can vary in size. The two parts are trained together end to end. On the FEVER benchmark, TwoWingOS clearly beats the previous best system on both evidence retrieval and verdict accuracy. This suggests that joint training helps both subtasks and is worth pursuing for fact-checking and for validating knowledge base facts.

Key points

  • Evaluation and baseline: Experiments use FEVER (Thorne et al., 2018). The baseline retrieves the top-5 sentences from the top-5 pages with TF-IDF and then applies a neural entailment classifier.
  • Test results against the prior top system:
    • Evidence F1: 47.15 vs. 17.47.
    • ScoreEv: 54.33 vs. 31.87.
    • NoScoreEv: 75.99 vs. 50.91.
    • The authors summarize this as roughly 30% better evidence F1 and roughly 23% higher verification accuracy.
  • Evidence selection: A sentence counts as evidence if its sigmoid probability exceeds 0.5, and this module is trained with binary cross-entropy. Candidate passages are retrieved using entity mentions found in the claim.
  • Claim verification representation: On the dev set, a two-channel fine-grained representation for claim verification added +7.42% NoScoreEv and +3% ScoreEv.
  • Error analysis:
    • Recall falls as gold evidence sets grow larger, while precision stays stable.
    • Unknown tokens with random embeddings cause mistakes.
    • Some claims require combining facts from several sentences or comparing numbers (e.g., 1983 vs. 1977).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

39 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 37 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1010 · arXiv 1808.03465

@inproceedings{yin2018twowingos,
  title = {{TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification}},
  author = {Yin, Wenpeng and Roth, Dan},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1010},
  eprint = {1808.03465},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d18-1010}
}

References

24 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jeff Pasternack and Dan Roth. 2013. Latent credibility analysis. In Proceedings of WWW, pages 1009– 1020. find
  2. Gabor Angeli and Christopher D. Manning. 2014. Nat-uralli: Natural logic inference for common sense reasoning. In Proceedings of EMNLP, pages 534– 545. find
  3. Benjamin Riedel, Isabelle Augenstein, Georgios P. Sp-ithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the fake news challenge stance detection task. CoRR, abs/1707.03264. find
  4. William Blacoe and Mirella Lapata. 2012. A comparison of vector-based representations for semantic composition. In Proceedings of EMNLP-CoNLL, pages 546–556. find
  5. Dan Roth, Mark Sammons, and V. G. Vinod Vydiswaran. 2009. A framework for entailed relation recognition. In Proceedings of ACL, pages 57–60. find
  6. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of SIGMOD, pages 1247–1250. find
  7. Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of WWW, pages 697–706. find
  8. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In Proceedings of NAACL. find our card
  9. Arun Tejasvi Chaganty, Ashwin Paranjape, Percy Liang, and Christopher D. Manning. 2017. Importance sampling for unbiased on-demand evaluation of knowledge base population. In Proceedings of EMNLP, pages 1038–1048. find
  10. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the Workshop on Language Technologies and Computational Social Science@ACL, pages 18–22. find our card
  11. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of ACL, pages 1870–1879. find our card
  12. V. G. Vinod Vydiswaran, ChengXiang Zhai, and Dan Roth. 2011. Content-driven trust propagation framework. In Proceedings of SIGKDD, pages 974–982. find
  13. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzoto. 2013. Recognizing textual entailment: Models and applications. find
  14. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018. R3 : Reinforced ranker-reader for open-domain question answering. In Proceedings of AAAI. find
  15. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121–2159. find
  16. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of NAACL, pages 1163–1168. find
  17. William Yang Wang. 2017. “Liar, Liar Pants on Fire”: A new benchmark dataset for fake news detection. In Proceedings of ACL, pages 422–426. find our card
  18. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard H. Hovy. 2013. Learning whom to trust with MACE. In Proceedings of NAACL, pages 1120–1130. find our card
  19. Wenpeng Yin, Sebastian Ebert, and Hinrich Schütze. 2016. Attention-based convolutional neural network for machine comprehension. In Proceedings of the NAACL Workshop on Human-Computer Question Answering, pages 15–21. find
  20. Mio Kobayashi, Ai Ishii, Chikara Hoshino, Hiroshi Miyashita, and Takuya Matsuzaki. 2017. Automated historical fact-checking by passage retrieval, word statistics, and virtual question-answering. In Proceedings of IJCNLP, pages 967–975. find
  21. Wenpeng Yin and Hinrich Schütze. 2017. Attentive convolution. CoRR, abs/1710.00519. — [arXiv:1710.00519](https://arxiv.org/abs/1710.00519) find
  22. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278–2324. find
  23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS, pages 3111–3119. find
  24. Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of EMNLP, pages 2249–2255. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yin-2018-twowingos-two-wing-optimization-strategy/