Elamur Research⌕
☰
home / papers / 2023 / AbLit: A Resource for Analyzing and Generating Abridged Versions of English Literature

AbLit: A Resource for Analyzing and Generating Abridged Versions of English Literature

Paper · 2023 · Melissa Roemmele, Kyle Shaffer, Katrina Olsen et al.

Summary

The paper treats abridgement as an NLP task for the first time. Abridgement means shortening a book while keeping its wording and style. It differs from abstractive summarization, which rewrites heavily, and from simplification, which usually works on single sentences. The authors build AbLit from ten public-domain English novels on Project Gutenberg, paired with free abridged versions written by one author, Emma Laybourn. Each chapter is aligned at sentence level using a dynamic-programming method adapted from the Wikipedia Simplification dataset. The method scores span pairs with ROUGE-1 precision and penalizes larger spans, so it prefers the narrowest alignment. Humans then check the alignments. Full checking is reserved for a test set; elsewhere, review targets only suspicious rows. The authors analyse how the abridged text relates to the original and test both extractive and generative models (T5, BART). The results show that abridgement is mostly a matter of extraction, but a hard one. Models keep original text well but are weak at choosing what to delete. The resource matters because it supports systematic study of shortening a text while staying faithful to it.

Key points

  • AbLit covers 10 British novels from the 18th and 19th centuries, split into 868 chapters (808 train, 10 dev, 50 test). Each abridgement is roughly half to two-thirds of the original length. Data and a Python package are at github.com/roemmele/AbLit.
  • The alignment settings are ROUGE-1 precision (abridged span as hypothesis), original spans of up to 3 sentences, abridged spans of 0–5 sentences, and a size penalty pn = 0.175. On the first-chapter assessment set (1,126 rows), the automatic rows reached F1 0.967 against gold.
  • Five validators corrected the assessment rows with very high agreement (Fleiss' κ = 0.984), spending 10–15 minutes per chapter.
  • In partial validation, only rows scoring below 0.9 were re-checked, and only if they had two or more abridged sentences or sat next to an empty-abridgement row. This raised F1 from 0.967 to 0.99, and the training set is estimated to be about 99% correct.
  • Evaluation separates three things: preserved words (Prsv), removed words (Rmv) and added words (Add).
    • ZERO SHOT T5 scores low on ROUGE-L, so AbLit differs from summarization data.
    • An oracle that removes exactly the right tokens (PERFECT EXT TOKS) scores high.
    • TUNED BART adds a few correct words but keeps more original text than the reference.

Interesting ideas

  • I1. Penalize alignment candidates by their span size, so the aligner picks the tightest one-to-one or one-to-few pairing over broad merged blocks; this works for any parallel-text alignment.
  • I2. Route human (or teacher) review only to automatic outputs that are both low-scoring and structurally risky, such as multi-sentence spans or neighbours of empty alignments; most of the quality gain comes at a small share of the effort.
  • I3. Split output quality into three separate scores, for what was kept, what was correctly removed and what was newly added, instead of one overlap number; this shows which of the three behaviours is failing.
  • I4. Use an oracle baseline that removes exactly the right tokens to show how much of the task is pure deletion; this separates the cost of choosing what to delete from the cost of producing fluent text.
  • I5. Build a fully validated test set while leaving the training set only partially validated, so evaluation stays strictly gold even when training data is silver.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): When checking the text → graph → text compressor, align output sentences to source sentences with a size-penalized DP, so every generated sentence traces back to a minimal source span.
  • I2 (learn): Silver trees where witness parsers disagree and that also match risky patterns (long coordination, attachments next to dropped tokens) could be sent to the LLM teacher tier instead of reviewing everything.
  • I3 (back): Score generated or compressed text separately on kept, dropped and invented content against the source, so hallucinated additions are visible apart from omissions.
  • I4 (answer): For shortened retellings of fables and tales, an oracle that deletes the right tree nodes would show whether errors come from choosing what to cut or from rewriting.
  • I5 (learn): Keep a small, fully hand-checked UD and frames test slice separate from the silver training pool, so evaluation numbers are not inflated by silver errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Melissa Roemmele, Kyle Shaffer, Katrina Olsen, Yiyi Wang and Steve DeNeefe. 2023. AbLit: A Resource for Analyzing and Generating Abridged Versions of English Literature. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.eacl-main.269

@inproceedings{roemmele2023ablit,
  title = {{AbLit: A Resource for Analyzing and Generating Abridged Versions of English Literature}},
  author = {Melissa Roemmele, Kyle Shaffer, Katrina Olsen, Yiyi Wang and Steve DeNeefe},
  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.eacl-main.269},
  url = {https://doi.org/10.18653/v1/2023.eacl-main.269}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Atef Chaudhury, Makarand Tapaswi, Seung Wook Kim, and Sanja Fidler. 2019. The shmoop corpus: A dataset of stories with loosely aligned summaries. arXiv preprint arXiv:1912.13082. ↗
  2. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. Advances in neural information processing systems, 32. find
  3. William Coster and David Kauchak. 2011. Simple English Wikipedia: A new text simplification task. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 665–669, Portland, Oregon, USA. Association for Computational Linguistics.
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Cristina Garbacea, Mengtian Guo, Samuel Carton, and Qiaozhu Mei. 2021. Explainable prediction of text complexity: The missing preliminaries for text simplification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1086–1097, Online. Association for Computational Linguistics. ↗
  6. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguistics. ↗
  7. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161–1166, Hong Kong, China. Association for Computational Linguistics. find
  8. Anna Kazantseva and Stan Szpakowicz. 2010. Summarizing short stories. Computational Linguistics, 36(1):71–109. ↗
  9. Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2021. Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08209. ↗
  10. Faisal Ladhak, Bryan Li, Yaser Al-Onaizan, and Kathleen McKeown. 2020. Exploring content selection in summarization of novel chapters. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5043–5054, Online. Association for Computational Linguistics. find
  11. Lynn Lauber. 1998. Bookend; confessions of (an) abridger. The New York Times. find
  12. Brittany Lavin. 2014. Abridgement: What it can mean for your book. Opyrus. find
  13. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. Dominican Republic. Association for Computational Linguistics.
  14. Vic Sussman. 1988. The fine art of abridgement. The Washington Post. find
  15. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862. ↗
  16. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. ↗
  17. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. ↗
  18. Weiwei Zhang, Jackie Chi Kit Cheung, and Joel Oren. 2019. Generating character descriptions for automatic summarization of fiction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7476–7483. find
  19. Ventsislav Zhechev. 2014. Analysing the post-editing of machine translation at autodesk. Post-editing of machine translation: Processes and Applications, pages 2–24. find
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  21. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. ↗
  22. Rada Mihalcea and Hakan Ceylan. 2007. Explorations in automatic book summarization. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 380–389, Prague, Czech Republic. Association for Computational Linguistics. find
  23. Duncan Minshull. 2001. The incredible shrinking book. The Guardian. find
  24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1– 67. ↗
  25. Renliang Sun, Hanqi Jin, and Xiaojun Wan. 2021. Document-level text simplification: Dataset, criteria and baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7997–8013, Online and Punta Cana, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/roemmele-2023-ablit-resource-analyzing-generating-abridged/