Elamur Research⌕
☰
home / papers / 2023 / PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition

PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition

Paper · 2023 · Sihao Chen, Senaka Buthpitiya, Alex Fabrikant et al.

Summary

Standard NLI datasets give one entailment label to a whole sentence or paragraph. Most real sentences pack several distinct units of meaning, and each can be true or unsupported independently. A single label therefore hides partial entailment, where only some of a hypothesis is backed by the premise. PropSegmEnt addresses this gap. It is a corpus of over 45K propositions built from clusters of topically aligned Wikipedia and news documents. Expert annotators split each sentence into propositions, each written as a subset of the sentence's tokens. They then labeled every proposition as entailed, neutral or contradicted with respect to another document from the same cluster. The authors train T5-Large seq2seq baselines for segmentation and entailment. They also show that combining proposition-level labels with logical AND, a pipeline they call PropNLI, stays stable on long DocNLI hypotheses where an MNLI model gets worse. The same setup gives more balanced token-level hallucination detection on summaries. The work matters because it makes entailment judgments fine-grained and explainable for compositional text.

Key points

  • About 90% of sentences that carry any informational content contain more than one proposition.
  • Between two related documents, 46% of sentences have some support. Of those, 74% are only partially entailed, so sentence-level NLI finds roughly a quarter of the meaningful links.
  • Propositions are token subsets rather than SRL or OpenIE predicate-argument tuples, because those formats miss propositions, do not resolve references and produce overly long arguments.
  • Documents are cut to their first ten sentences because annotation cost grows quadratically with the number of propositions. Contradictions are rare, which the authors attribute to uncertain reference determinacy.
  • In the DocNLI case study, PropNLI and a T5-Large MNLI model perform similarly below 100 tokens. As hypotheses grow longer, MNLI accuracy falls while PropNLI holds steady. MNLI training hypotheses average about 21.73 tokens.

Interesting ideas

  • I1. Represent a proposition as a selection of the sentence's own tokens, possibly discontinuous, instead of a typed tuple, so the granularity of each argument can vary freely.
  • I2. Judge each atomic claim separately and then combine the results with logical conjunction, so a single unsupported piece cannot be hidden by a sentence-level verdict.
  • I3. Treat partial entailment as its own outcome rather than merging it into non-entailment, since this hidden category turned out to be three times larger than full entailment.
  • I4. Bucket the evaluation set by input length to expose where a model trained on short inputs breaks down.
  • I5. Treat decontextualization as part of the proposition, so that "his hometown" resolves to Andy Warhol and Pittsburgh.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Frames from prag could be tied back to the exact token sets that justify them, giving each frame a transparent textual anchor and revealing propositions the VerbNet roles fail to cover.
  • I2 (answer): Readers answering fable or school-test questions could check each proposition of a candidate answer against the world of the text and report which part fails.
  • I3 (world): When comparing two texts or a text against the knowledge graph, Mova could mark facts as supported, unsupported or mixed per proposition, avoiding all-or-nothing merges.
  • I4 (learn): Evaluations of parsers and readers could be split by sentence or passage length so that length-driven failures appear explicitly next to the negative controls.
  • I5 (frames): Coreference output could be folded into each extracted frame so that every proposition stands alone and can be checked in isolation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Dan Roth and Tal Schuster. 2023. PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.565

@inproceedings{chen2023propsegment,
  title = {{PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition}},
  author = {Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Dan Roth and Tal Schuster},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.565},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.565}
}

References

43 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. TensorFlow: a system for Large-Scale machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265–283. 8882 9 ↗
  2. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The Berkeley FrameNet project. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics. ↗
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  4. Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. 2022. Generating literal and implied sub-questions to fact-check complex claims. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3495–3516, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  5. Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021. Improving faithfulness in abstractive summarization with contrast candidate generation and selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5935–5941, Online. Association for Computational Linguistics.
  6. Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making sentences stand-alone. Transactions of the Association for Computational Linguistics, 9:447–461. ↗
  7. Lei Cui, Furu Wei, and Ming Zhou. 2018. Neural open information extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 407–413, Melbourne, Australia. Association for Computational Linguistics.
  8. I. Dagan and O. Glickman. 2004. Probabilistic textual entailment: Generic applied modeling of language variability. In Learning Methods for Text Understanding and Mining. find
  9. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine learning challenges workshop, pages 177–190. Springer. find
  10. Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In Proceedings of ACL-08: HLT, pages 1039–1047, Columbus, Ohio. Association for Computational Linguistics.
  11. Luciano Del Corro and Rainer Gemulla. 2013. ClausIE: clause-based open information extraction. In Proceedings of the 22nd international conference on World Wide Web, pages 355–366. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  13. Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open information extraction from the web. Communications of the ACM, 51(12):68–74. ↗
  14. Farima Fatahi Bayat, Nikita Bhutani, and H. Jagadish. 2022. CompactIE: Compact facts in open information extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 900–910, Seattle, United States. Association for Computational Linguistics. find
  15. Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051–2060, Melbourne, Australia. Association for Computational Linguistics.
  16. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. ↗
  17. Kiril Gashteovski, Rainer Gemulla, Bhushan Kotnis, Sven Hertling, and Christian Meilicke. 2020. On aligning OpenIE extractions with knowledge bases: A case study. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 143–154, Online. Association for Computational Linguistics. find
  18. Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3592–3603, Online. Association for Computational Linguistics.
  19. Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, You Wu, Cong Yu, Daniel Finnie, Hongkun Yu, Jiaqi Zhai, and Nicholas Zukoski. 2020. Generating representative headlines for news stories. In Proceedings of The Web Conference 2020, pages 1773– 1784. find
  20. Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 643–653, Lisbon, Portugal. Association for Computational Linguistics. ↗
  21. Paul R Kingsbury and Martha Palmer. 2002. From TreeBank to PropBank. In LREC, pages 1989–1993. 8883 10
  22. Ayal Klein, Eran Hirsch, Ron Eliav, Valentina Pyatkin, Avi Caciularu, and Ido Dagan. 2022. QASem parsing: Text-to-text modeling of QA-based semantics. arXiv preprint arXiv:2205.11413. ↗
  23. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Computational Linguistics.
  24. Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97. ↗
  25. Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. SummaC: Re-visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177. ↗
  26. Omer Levy, Torsten Zesch, Ido Dagan, and Iryna Gurevych. 2013. Recognizing partial textual entailment. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 451–455, Sofia, Bulgaria. Association for Computational Linguistics. find
  27. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. ↗
  28. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The NomBank project: An interim report. In Proceedings of the workshop frontiers in corpus annotation at hlt-naacl 2004, pages 24–31.
  29. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics. ↗
  30. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational linguistics, 31(1):71–106. ↗
  31. Martha Palmer, Daniel Gildea, and Nianwen Xue. 2010. Semantic role labeling. Synthesis Lectures on Human Language Technologies, 3(1):1–103. ↗
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67. ↗
  33. Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. 2022. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203.17189. ↗
  34. Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer, and Ido Dagan. 2020. Controlled crowdsourcing for high-quality QA-SRL annotation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7008–7013, Online. Association for Computational Linguistics. ↗
  35. Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, and Donald Metzler. 2022. Stretching sentence-pair NLI models to reason over long documents and clusters. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 394–412, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  36. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643, Online. Association for Computational Linguistics.
  37. Joe Stacey, Pasquale Minervini, Haim Dubossarsky, and Marek Rei. 2022. Logical reasoning with span predictions: Span-level logical atoms for interpretable and robust nli models. In The Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  38. Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 885– 895, New Orleans, Louisiana. Association for Computational Linguistics. find
  39. Asher Stern and Ido Dagan. 2014. Recognizing implied predicate-argument relationships in textual inference. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 739–744, Baltimore, Maryland. Association for Computational Linguistics. find
  40. Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order matters: Sequence to sequence for sets. In Proceedings of the International Conference on Learning Representations. ↗
  41. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform 8884 11 for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019. find
  42. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  43. Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. DocNLI: A large-scale dataset for document-level natural language inference. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4913–4922, Online. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-propsegment-large-scale-corpus-proposition/