Elamur Research⌕
☰
home / papers / 2019 / Plain English Summarization of Contracts

Plain English Summarization of Contracts

Paper · 2019 · Laura Manor and Junyi Jessy Li

Summary

Few people read terms of service or similar one-sided contracts, mainly because these documents are long and hard to understand. The authors propose a new task: summarizing contract sections in plain English for non-lawyers. They scraped section-level pairs of contract text and community-written summaries from two sites, TL;DRLegal and TOS;DR. Drawing on three years spent drafting contracts, the first author screened the summaries and kept the most informative one per set. The result is a small evaluation set, 446 sets in total, released on GitHub. The analysis shows two things. First, the reference summaries contain many more novel n-grams than DUC 2002 news summaries. Second, readability metrics place the summaries about six grade levels below the source text. Unsupervised extractive methods (TextRank, Greedy KL, Lead-K) score much worse here than on DUC 2002. The task therefore needs real abstraction, simplification and register change, not sentence selection. The authors call for unsupervised simplification and style-transfer resources for legal language.

Key points

  • Motivation: a cited study found up to 98% of users skip reading terms of service in full. In the paper's example, a 78-word Pokemon GO clause is condensed into 19 words.
  • Data sources:
    • TL;DRLegal: 84 sets from 9 company documents, including YouTube, Facebook, Dropbox and Minecraft.
    • TOS;DR: 421 sets from 166 documents by 122 companies, each with 1–3 summaries.
  • Manual curation:
    • 361 sets kept at least one good summary. Of the chosen summaries, 152 were "templatic" (111 of them unique).
    • 369 summaries were rejected: 291 duplicates, 63 verbatim copies, 24 with opinions not inferable from the source, and 20 that described only the topic. Summaries longer than their source were also dropped.
  • Abstraction and readability: the dataset has far more novel n-grams than DUC 2002 and an average gap of about 6 reading-grade levels.
  • Baselines: TextRank, Greedy KL (KLSum) and Lead-K do clearly worse than on news. The data is meant for evaluation, not supervised training.

Interesting ideas

  • I1. The degree of abstraction is quantified by counting reference n-grams absent from the source and comparing against a familiar news benchmark. This gives a cheap, interpretable measure of how far a rewrite drifts from source wording.
  • I2. Noisy crowd references are filtered with explicit, countable rejection reasons: duplicate, verbatim copy, unsupported opinion, topic-only, or longer than the source. Each removal is auditable rather than a vague quality score.
  • I3. Readability formulas measure simplification as a grade-level gap between source and output, separately from content overlap.
  • I4. Summaries are aligned per section rather than per document, so a reader can always return to the exact detailed clause behind a short statement.
  • I5. Many accepted summaries repeat across services with only the company name changed. This suggests contracts are built from a limited set of recurring clause types.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): the text → graph → text compressor could report its novel n-gram rate against a reference corpus, showing whether generation rephrases meaning or just copies surface strings.
  • I2 (learn): the same reason-coded filters could screen LLM-teacher outputs. The "not inferable from source" check could be implemented as a test against the world of the text.
  • I3 (back): when generating answers or simplified paraphrases, a source-vs-output grade-level difference would serve as a second, style-focused evaluation axis next to semantic fidelity.
  • I4 (answer): each answer could carry a pointer to the source span that justifies it, which fits the requirement that every step be explainable.
  • I5 (frames): one could define a small frame inventory for contract clauses (granting permission, terminating an account, indemnifying a party), with roles filled from the UD tree.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States109 works · 56,942 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSummarization · 232 worksLegal AI · 134 works

Cite

Cite as Laura Manor and Junyi Jessy Li. 2019. Plain English Summarization of Contracts. Proceedings of the Natural Legal Language Processing Workshop 2019.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-2201

@inproceedings{manor2019plain,
  title = {{Plain English Summarization of Contracts}},
  author = {Laura Manor and Junyi Jessy Li},
  booktitle = {Proceedings of the Natural Legal Language Processing Workshop 2019},
  year = {2019},
  doi = {10.18653/v1/w19-2201},
  url = {https://doi.org/10.18653/v1/w19-2201}
}

References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol 1: Long Papers), pages 675–686.
  2. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. In Technical Report, Institute for Simulation and Training, University of Central Florida. ↗
  3. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283. ↗
  4. Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865–1874. ↗
  5. Atefeh Farzindar and Guy Lapalme. 2004. Legal text summarization by exploration of the thematic structure and argumentative roles. Text Summarization Branches Out. find
  6. Filippo Galgani, Paul Compton, and Achim Hoffmann. 2012. Combining different summarization techniques for legal text. In Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data, pages 115–123. find
  7. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
  8. G Harry Mc Laughlin. 1969. Smog grading-a new readability formula. Journal of reading, 12(8):639– 646. find
  9. General Data Protection Regulation. 2018. Regulation on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (data protection directive). L119, 4 May 2016, pages 188. find
  10. David Mellinkoff. 2004. The language of the law. Wipf and Stock Publishers. find
  11. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing. find
  12. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830–6841. ↗
  13. Shashi Narayan and Claire Gardent. 2016. Unsupervised sentence simplification using deep semantics. In The 9th International Natural Language Generation conference, pages 111–120. ↗
  14. Lucia Specia. 2010. Translating from complex to simplified sentences. In Proceedings of the International Conference on Computational Processing of the Portuguese Language, pages 30–39. find
  15. Michael E Sykuta, Peter G Klein, and James Cutts. 2007. Cori k-base: Data overview. ↗
  16. Ani Nenkova, Kathleen McKeown, et al. 2011. Automatic summarization. Foundations and Trends R in Information Retrieval, 5(2–3):103–233. find
  17. TAC. 2014. In https://tac.nist.gov/2014/BiomedSumm/. ↗
  18. Benjamin Nye and Ani Nenkova. 2015. Identification and characterization of newsworthy verbs in world news. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1440–1445. find
  19. Jie Tang, Bo Wang, Yang Yang, Po Hu, Yanting Zhao, Xinyu Yan, Bo Gao, Minlie Huang, Peng Xu, Weichang Li, et al. 2012. Patentminer: topic-driven patent analysis and mining. In Proceedings of the 18th Internationasl Conference on Knowledge Discovery and Data Mining, pages 1366–1374. find
  20. Jonathan A Obar and Anne Oeldorf-Hirsch. 2018. The biggest lie on the internet: Ignoring the privacy policies and terms of service policies of social networking services. Information, Communication & Society, pages 1–20. find
  21. Peter M Tiersma. 2000. Legal language. University of Chicago Press. find
  22. Yuen-Hsien Tseng, Chi-Jen Lin, and Yu-I Lin. 2007. Text mining techniques for patent analysis. Information Processing & Management, 43(5):1216– 1247. ↗
  23. Paul Over, Hoa Dang, and Donna Harman. 2007. DUC in context. Information Processing & Management, 43(6):1506–1520. ↗
  24. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of the conference on empirical methods in natural language processing, pages 409–420.
  25. Gustavo H. Paetzold and Lucia Specia. 2016. Unsupervised lexical simplification for non-native speakers. In Proceedings of the 13th Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence, pages 3761–3767. find
  26. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. ↗
  27. Plain English law. 1978. Title 7: Requirements for use of plain language in consumer transactions. The Laws Of New York Consolidated Laws. General Obligations. Article 5: Creation, Definition And Enforcement Of Contractual Obligations. find
  28. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415. ↗
  29. Plain Writing Act. 2010. An act to enhance citizen access to government information and services by establishing that government documents issued to the public must be written clearly, and for other purposes. House of Representatives 946; Public Law No. 111-274; 124 Statues at Large 2861. find
  30. Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. 2018. Unsupervised text style transfer using language models as discriminators. In Advances in Neural Information Processing Systems, pages 7298–7309. ↗
  31. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389.
  32. Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri Irene Li Dan, and Friedman Dragomir R Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In Proceedings of the 13th Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence. find
  33. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol 1: Long Papers), pages 1073–1083. ↗
  34. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584–594.
  35. RJ Senter and Edgar A Smith. 1967. Automated readability index. Technical report, Cincinnati Univ OH. find
  36. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd international conference on computational linguistics, pages 1353–1361.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/manor-2019-plain-english-summarization-contracts/