Elamur Research⌕
☰
home / papers / 2020 / From Arguments to Key Points: Towards Automatic Argument Summarization

From Arguments to Key Points: Towards Automatic Argument Summarization

Paper · 2020 · Roy Bar-Haim, Lilach Eden, Roni Friedman et al.

Summary

The paper addresses how to condense hundreds or thousands of pro and con arguments on a controversial topic into something a decision maker can actually read. The authors propose a summary made of a few short "key points" (high-level talking points), each weighted by how many arguments it covers. They drew 7,000 crowd-sourced arguments on 28 topics from IBM-Rank-30k. A professional debater wrote up to seven key points per side for each topic without seeing the arguments. Annotation showed that these small sets cover most arguments, so an expert can often anticipate the main points in advance. Treating the key points as given, the authors frame the task as argument-to-key point mapping and release ArgKP, the first large dataset for it, with roughly 24,000 labeled pairs. In cross-topic experiments, unsupervised embedding similarity performed poorly. A fine-tuned BERT-large match scorer combined with a suitable selection policy gave promising results. This work is an intermediate step toward fully automatic argument summarization.

Key points

  • Arguments come from IBM-Rank-30k. The authors kept the 28 topics that also appear as motions in iDebate's Debatabase, discarded arguments with quality below 0.5 or polarity confidence below 0.6, and sampled 250 arguments per topic.
  • The headlines of Debatabase points were rejected as key points because their specificity varied widely and they covered few crowd arguments. A debater instead wrote at most 7 key points per side under time limits.
  • ArgKP contains about 24,000 (argument, key point) pairs labeled match/no-match, covering more than 6,500 arguments after filtering. Previous related datasets covered only 2–4 topics.
  • BERT-large fine-tuned on ArgKP was the best match scorer. With the Dual Threshold selection policy it reached F1 = 0.73 and accuracy ≈ 0.887 over all arguments.
  • Four selection policies were compared: Threshold, Best Match, BM+Threshold and Dual Threshold.
    • Best Match works well when an argument has exactly one matching key point.
    • It fails when an argument has no match or several matches.
    • BM+Threshold gives the highest precision but low recall.

Interesting ideas

  • I1. Summaries as counted talking points. Representing a large opinion collection as a short list of statements plus how many items match each one gives both a readable and a quantitative view. This could suit survey responses, product reviews or public consultations.
  • I2. Expert-written labels before seeing the data. Having a domain expert draft the category set blind, then checking its coverage, is a cheap way to build a label inventory. The share of unmatched items then becomes a natural novelty signal.
  • I3. Separate scoring from selection. The design scores every (item, candidate) pair independently and then applies a selection policy to decide which candidates to assign. Analyzing the policies separately for items with zero, one or several gold matches shows each policy's failure mode. This is useful for any multi-label assignment task with an optional "none" outcome.
  • I4. Treat matching as NLI-like pair classification. The argument/key point relation resembles text/hypothesis in entailment. Supervised cross-encoders clearly beat plain embedding similarity here, which suggests cosine similarity is insufficient for fine-grained semantic matching.
  • I5. Cross-topic evaluation as the default. Training and testing on different topics measures whether a matcher generalizes to new debates. This matters wherever new topics or domains keep appearing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States85 works · 66,118 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Summarization · 159 works

Cite

Cite as Roy Bar-Haim, Lilach Eden, Roni Friedman, Yoav Kantor, Dan Lahav and Noam Slonim. 2020. From Arguments to Key Points: Towards Automatic Argument Summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.371

@inproceedings{barhaim2020arguments,
  title = {{From Arguments to Key Points: Towards Automatic Argument Summarization}},
  author = {Roy Bar-Haim, Lilach Eden, Roni Friedman, Yoav Kantor, Dan Lahav and Noam Slonim},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.371},
  url = {https://doi.org/10.18653/v1/2020.acl-main.371}
}

References

34 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yamen Ajjour, Milad Alshomary, Henning Wachsmuth, and Benno Stein. 2019. Modeling frames in argumentation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2922–29… find
  2. Roy Bar-Haim, Indrajit Bhattacharya, Francesco Din-uzzo, Amrita Saha, and Noam Slonim. 2017. Stance classification of context-dependent claims. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 251–261. Associ… our card
  3. Filip Boltužić and Jan Šnajder. 2014. Back up your stance: Recognizing arguments in online discussions. In Proceedings of the First Workshop on Argumentation Mining, pages 49–58, Baltimore, Maryland. Association for Computational Linguistics. find
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… find
  5. Elena Cabrio and Serena Villata. 2013. A natural language bipolar argumentation approach to support users in online debate interactions. Argument & Computation, 4(3):209–230. find
  6. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. our card
  7. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364. our card
  8. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  10. Charlie Egan, Advaith Siddharthan, and Adam Wyner. 2016. Summarising the points made in online political debates. In Proceedings of the Third Workshop on Argument Mining (ArgMining2016), pages 134–143, Berlin, Germany. Association for Computational Linguistics. find
  11. Liat Ein-Dor, Eyal Shnarch, Lena Dankin, Alon Halfon, Benjamin Sznajder, Ariel Gera, Carlos Alzate, Martin Gleize, Leshem Choshen, Yufang Hou, Yonatan Bilu, Ranit Aharonov, and Noam Slonim. 2020. Corpus wide argument mining – a working solution. In AAAI. find
  12. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
  13. Shai Gretz, Roni Friedman, Edo Cohen-Karlik, Assaf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slonim. 2020. A large-scale dataset for argument quality ranking: Construction and analysis. In AAAI. find
  14. Kazi Saidul Hasan and Vincent Ng. 2014. Why are you taking this stance? identifying and classifying reasons in ideological debates. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 751– 762, Doha, Qatar. Association for Computational Linguistic… our card
  15. J. R. Landis and G. G. Koch. 1997. The measurements of observer agreement for categorical data. Biometrics, 33:159–174. find
  16. John Lawrence and Chris Reed. 2020. Argument mining: A survey. Computational Linguistics, 45(4):765–818. our card
  17. Ran Levy, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni, and Noam Slonim. 2014. Context dependent claim detection. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1489– 1500, Dublin, Ireland. Dublin City University and Associati… our card
  18. Ran Levy, Ben Bogin, Shai Gretz, Ranit Aharonov, and Noam Slonim. 2018. Towards an argumentative content search engine using weak supervision. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2066–2081. Association for Computational Linguistics. find
  19. Ran Levy, Shai Gretz, Benjamin Sznajder, Shay Hummel, Ranit Aharonov, and Noam Slonim. 2017. Unsupervised corpus–wide claim detection. In Proceedings of the 4th Workshop on Argument Mining, pages 79–84. Association for Computational Linguistics. find
  20. Amita Misra, Brian Ecker, and Marilyn Walker. 2016. Measuring the similarity of sentential arguments in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 276–287, Los Angeles. Association for Computational Linguistics. find
  21. Marie-Francine Moens, Erik Boiy, Raquel Mochales Palau, and Chris Reed. 2007. Automatic detection of arguments in legal texts. In Proceedings of the 11th International Conference on Artificial Intelligence and Law, ICAIL ’07, pages 225–230, New York, NY, USA. ACM. find
  22. Nona Naderi. 2016. Argumentation mining in parliamentary discourse. In Proceedings of the 11th International Conference of the Ontario Society for the Study of Argumentation, pages 1–9. find
  23. Joonsuk Park and Claire Cardie. 2014. Identifying appropriate support for propositions in online user comments. In Proceedings of the First Workshop on Argumentation Mining, pages 29–38, Baltimore, Maryland. Association for Computational Linguistics. and methods. In Proceedings of the 2019 Conferenc… our card
  24. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. our card
  25. Henning Wachsmuth, Martin Potthast, Khalid Al Khatib, Yamen Ajjour, Jana Puschmann, Jiani Qu, Jonas Dorsch, Viorel Morari, Janek Bevendorff, and Benno Stein. 2017. Building an argument search engine for the web. In Proceedings of the 4th Workshop on Argument Mining, pages 49–59. Association for Comp… find
  26. Isaac Persing and Vincent Ng. 2016. End-to-end argumentation mining in student essays. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1384–1394, San Diego, California. Association for Computatio… our card
  27. Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2019. Classification and clustering of arguments with contextualized word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 567– 578, Fl… our card
  28. Ruty Rinott, Lena Dankin, Carlos Alzate Perez, Mitesh M. Khapra, Ehud Aharoni, and Noam Slonim. 2015. Show me your evidence - an automatic method for context dependent evidence detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 440–450, Lisbon… our card
  29. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  30. Adam Wyner, Raquel Mochales-Palau, Marie-Francine Moens, and David Milward. 2010. Approaches to text mining arguments from legal cases. In Semantic Processing of Legal Texts: Where the Language of Law Meets the Law of Language, page 60–79, Berlin, Heidelberg. Springer-Verlag. find
  31. Christian Stab, Johannes Daxenberger, Chris Stahlhut, Tristan Miller, Benjamin Schiller, Christopher Tauchmann, Steffen Eger, and Iryna Gurevych. 2018a. Argumentext: Searching for arguments in heterogeneous sources. In Proceedings of the 2018 Conference of the North American Chapter of the Associati… find
  32. Christian Stab and Iryna Gurevych. 2017. Parsing argumentation structures in persuasive essays. Computational Linguistics, 43(3):619–659. our card
  33. Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych. 2018b. Cross-topic argument mining from heterogeneous sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3664–3674. Association for Computational Linguistics. our card
  34. Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, and Noam Slonim. 2019. Automatic argument quality assessment - new datasets find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haim-2020-arguments-key-points-automatic-argument/