Elamur Research⌕
☰
home / papers / 2021 / Paragraph-level Simplification of Medical Texts

Paragraph-level Simplification of Medical Texts

Summary

The paper addresses automatic simplification of biomedical writing. Reliable clinical evidence is usually written in heavy jargon, and manual rewriting cannot keep pace with the literature. Before this work, no large paragraph-level resource existed for the task. The authors scraped the Cochrane Database of Systematic Reviews, which pairs technical abstracts with author-written plain-language summaries (PLS). Using heuristics, they kept only the results-and-conclusions portions that roughly correspond across the two texts, which yielded 4459 aligned pairs.

They show that classic readability formulas barely separate the two styles. A much clearer separation comes from a score built on how likely each token is under a masked language model that was pretrained on scientific writing.

For generation, they fine-tune BART and add an unlikelihood penalty that discourages the decoder from emitting "technical" tokens. This makes outputs more readable and more abstractive, with only a small loss in content quality. However, strong penalties cause hallucinations, misspellings and repetition.

The work matters because it provides a public benchmark and a practical way to measure and reduce technicality in health communication.

Key points

  • Dataset construction:
    • 7820 abstract–PLS pairs were scraped from the Cochrane Library.
    • The authors kept only pairs whose texts fit within 1024 tokens (BART's input limit) and whose length ratio was between 0.2 and 1.3.
    • The final corpus has 4459 pairs. Code and data are released on GitHub.
  • Alignment heuristic:
    • For abstracts, text was kept from the Main Results section onward.
    • For PLS, the starting point was found with keyword triggers (e.g., "found", "evidence", "study characteristic", "trial").
    • About 56% of PLS are unsectioned "long-form" texts, which needed a separate rule.
  • Readability findings: Flesch-Kincaid and Automated Readability Index show only small gaps between abstracts and PLS. The MLM-likelihood metric captures the difference far better.
  • Output length:
    • Average lengths are 492 tokens for abstracts and 255 for reference PLS.
    • BART without unlikelihood produces about 228 tokens. UL-Cochrane produces about 164, UL-Newsela 201, and UL-Both 174.
    • Unlikelihood training also lowers n-gram overlap with the source, and the drop grows with n.
  • Failure modes:
    • Penalty strengths of α ≥ 1000 severely hurt coherence.
    • A frequent hallucination is an invented sentence of the form "evidence is current to [date]".
    • Drug and disease names are the words most often misspelled.

Interesting ideas

  • I1. Mine pairs from loosely parallel documents. When two document versions are not sentence-aligned, keep only the sections that line up, located with simple header and keyword rules. This kind of cheap structural alignment could produce parallel data in other domains, such as legal or government texts for Ukrainian.
  • I2. Measure style with a masked LM. Score technicality by how probable a text's tokens are under a masked LM trained on specialist text. This captures vocabulary and register shifts that length-based readability formulas miss. It could be reused to evaluate simplification or domain style without supervised readability labels.
  • I3. Use unlikelihood loss with learned word weights. Instead of hand-picking a jargon list, the penalized vocabulary comes from a logistic regression that discriminates the two styles. This is a general, data-driven way to steer a generator away from unwanted lexical classes, such as Russianisms, bureaucratese or slang.
  • I4. Check that penalty weights fit the domain. Weights learned on Newsela had a weaker effect than in-domain Cochrane weights. The Cochrane weights penalized numbers and statistics, which in turn reduced copying from the source. Any vocabulary-level control is worth inspecting this way.
  • I5. Diagnose systematic artifacts in the training data. The "evidence is current to" hallucination comes from target-side content that is absent from the source. Checking for target-only boilerplate before training can predict such hallucinations in any summarization dataset.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 60 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.395

@inproceedings{devaraj2021paragraph,
  title = {{Paragraph-level Simplification of Medical Texts}},
  author = {Devaraj, Ashwin and Marshall, Iain and Wallace, Byron and Li, Junyi Jessy},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.395},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.395}
}

References

56 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emil Abrahamsson, Timothy Forni, Maria Skeppstedt, and Maria Kvist. 2014. Medical text simplification using synonym replacement: Adapting assessment of word difficulty to a compounding language. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), pages 57–65. find
  2. Viraj Adduru, Sadid A Hasan, Joey Liu, Yuan Ling, Vivek V Datla, Ashequl Qadir, and Oladimeji Farri. 2018. Towards dataset creation and establishing baselines for sentence-level neural clinical paraphrase generation and simplification. In KHD@ IJ-CAI. find
  3. Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2020. Data-driven sentence simplification: Survey and benchmark. Computational Linguistics, 46(1):135–187. ↗ our card
  4. Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, et al. 2018. Construction of the literature graph in semantic scholar. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 84–91. ↗
  5. Paul W Armstrong and C David Naylor. 2019. Counteracting health misinformation: a role for medical journals? Jama, 321(19):1863–1864. find
  6. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3606– 3611. find
  7. Jeanne Sternlicht Chall. 1958. Readability: An appraisal of research and application. 34. Ohio State University. find
  8. Iain Chalmers, Douglas G Altman, et al. 1995. Systematic reviews. BMJ Publishing London. find
  9. Cochrane. 2013. Standards for the reporting of plain language summaries in new cochrane intervention reviews (pleacs). find
  10. William Coster and David Kauchak. 2011. Simple English Wikipedia: a new text simplification task. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 665–669. find our card
  11. Jerwin Jan S Damay, Gerard Jaime D Lojico, Kimberly Amanda L Lu, D Tarantan, and E Ong. 2006. SIMTEXT: Text simplification of medical literature. In Proceedings of the 3rd National Natural Language Processing Symposium-Building Language Tools and Resources, pages 34–38. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. ↗
  13. Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3393–3402. Susannah Health Fox and Maeve online 2013. ↗ our card
  14. Duggan. 2013. https://www.pewresearch.org/internet/2013/01/15/ health-online-2013/ . Online; accessed April 2, 2021. ↗
  15. Chris Gledhill, Hanna Martikainen, Alexandra Mestivier, and Maria Zimina. 2019. Towards a linguistic definition of ‘simplified medical english’: Applying textometric analysis to cochrane medical abstracts and their plain language versions. LCM - La Collana / The Series, 9788879169196:91–114. find
  16. Natalia Grabar and Rémi Cardon. 2018. CLEAR – simple corpus for medical French. In Proceedings of the 1st Workshop on Automatic Text Adaptation (ATA), pages 3–9. find
  17. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1693–1701. Curran Associates, Inc. ↗ our card
  18. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations. ↗
  19. John PA Ioannidis, Michael E Stuart, Shannon Brownlee, and Sheri A Strite. 2017. How to survive the medical misinformation mess. European journal of clinical investigation, 47(11):795–802. ↗
  20. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657. find our card
  21. Antonia Jelicic Kadic, Mahir Fidahic, Milan Vujcic, Frano Saric, Ivana Propadalo, Ivana Marelja, Svjetlana Dosenovic, and Livia Puljak. 2016. Cochrane plain language summaries are highly heterogeneous with low adherence to the standards. BMC medical research methodology, 16(1):61. ↗
  22. Sasikiran Kandula, Dorothy Curtis, and Qing Zeng-Treitler. 2010. A semantic and syntactic text simplification tool for health content. In AMIA annual symposium proceedings, pages 366–370. find
  23. Jasna Karačić, Pierpaolo Dondio, Ivan Buljan, Darko Hren, and Ana Marušić. 2019. Languages for different health information readers: multitrait-multimethod content analysis of cochrane systematic reviews textual summary formats. BMC medical research methodology, 19(1):75. ↗
  24. Rohit Kate, Xiaoqiang Luo, Siddharth Patwardhan, Martin Franz, Radu Florian, Raymond Mooney, Salim Roukos, and Chris Welty. 2010. Learning to predict readability using diverse linguistic features. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 546–554. find
  25. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research Branch. ↗
  26. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. ↗
  27. George Roger Klare. 1963. Measurement of readability. find
  28. Reno Kriz, João Sedoc, Marianna Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, and Chris Callison-Burch. 2019. Complexity-weighted loss and diverse reranking for sentence simplification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3137– 3147. ↗
  29. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880. find our card
  30. Junyi Jessy Li and Ani Nenkova. 2015. Fast and accurate prediction of sentence specificity. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2281–2287. find
  31. Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020. Don’t say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715– 4728. find
  32. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. find
  33. Leonardo Campillos Llanos, Dhouha Bouamor, Pierre Zweigenbaum, and Sophie Rosset. 2016. Managing linguistic and terminological variation in a medical dialogue system. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3167–3173. find
  34. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. International Conference on Learning Representations. find
  35. Partha Mukherjee, Gondy Leroy, David Kauchak, Srinidhi Rajanarayanan, Damian Y Romero Diaz, Nicole P Yuan, T Gail Pritchard, and Sonia Colina. 2017. Negait: A new parser for medical text simplification using morphological, sentential and double negation. Journal of biomedical informatics, 69:55– 62. ↗
  36. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807. ↗
  37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗ our card
  38. Sarah E Petersen and Mari Ostendorf. 2007. Text simplification for language learners: a corpus analysis. In Workshop on Speech and Language Technology in Education. ↗
  39. RJ Senter and Edgar A Smith. 1967. Automated readability index. Technical report, CINCINNATI UNIV OH. find
  40. Advaith Siddharthan. 2014. A survey of research on text simplification. ITL-International Journal of Applied Linguistics, 165(2):259–298. ↗
  41. Neha Srikanth and Junyi Jessy Li. 2020. Elaborative simplification: Content addition and explanation generation in text simplification. arXiv preprint arXiv:2010.10035. ↗
  42. Sowmya Vajjala and Ivana Lučić. 2018. Onestopenglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications, pages 297–304. find our card
  43. Laurens Van den Bercken, Robert-Jan Sips, and Christoph Lofi. 2019. Evaluating neural text simplification in the medical domain. In The World Wide Web Conference, pages 3286–3292. find
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. find
  45. Tong Wang, Ping Chen, John Rochford, and Jipeng Qiang. 2016. Text simplification using neural machine translation. In Thirtieth AAAI Conference on Artificial Intelligence. find
  46. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019. Neural text generation with unlikelihood training. In International Conference on Learning Representations. ↗ our card
  47. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗ our card
  48. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 409–420. find our card
  49. World Health Organization. 2013. Health literacy: the solid facts. World Health Organization. Regional Office for Europe. find
  50. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1015– 1024. find our card
  51. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. ↗ our card
  52. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415. ↗ our card
  53. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584–594. find our card
  54. Yang Zhong, Chao Jiang, Wei Xu, and Junyi Jessy Li. 2020. Discourse level factors for sentence deletion in text simplification. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  55. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). find our card
  56. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1353–1361. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/devaraj-2021-paragraph-level-simplification-medical-texts/