Elamur Research⌕
☰
home / papers / 2026 / Evaluation of Multilingual Text Simplification for the Mental Health Domain: Exploring Small Language Models

Evaluation of Multilingual Text Simplification for the Mental Health Domain: Exploring Small Language Models

Summary

The paper looks at how to judge automatic simplification of mental health texts in languages other than English, where such work is scarce. The authors took five ICD-11 articles on primary psychotic disorders (6A20–6A24, with sub-definitions) and simplified them in English, German and French. They used nine small models: LingConv, PRISM, six Ascle variants and a quantized BioMistral 7B that was prompted to rewrite the text for laypeople. Rather than relying on similarity scores like BLEU, ROUGE and BERTScore, which barely separated good from bad outputs, they measured how much lexical complexity and readability changed between the source and each output. Manual inspection of the English results, plus a small human ranking of BioMistral runs, showed that MTLD and FRE together were the most informative pair. Only BioMistral produced usable simplifications, and only in English. German and French outputs stayed dense or contained grammatical errors. The authors call the study preliminary but suggest that the combined MTLD/FRE shift could serve as a cheap language-agnostic check before human review.

Key points

  • Data: ICD-11 definitions 6A20–6A24, retrieved via the ICD API. Models: LingConv (T5-Base), PRISM (39 languages), Ascle (BigBird-Pegasus Large, BART Large and BioBART v2 Base, each in two fine-tuned versions) and BioMistral 7B SLERP with 4-bit GPTQ.
  • Metrics: 3 lexical complexity measures (TTR, MATTR, MTLD) and 9 readability indices (FRE, FKGL, ARI, SMOG, Fog, CLI, DCR, LIX, RIX), computed with textstat and LexicalRichness. Each was scored as a difference between the original and the simplified text, which makes cross-language comparison easier.
  • Interpreting the two key metrics:
    • Good simplifications raised both MTLD and FRE.
    • A rise in MTLD combined with a drop in FRE signals overly dense text.
    • A drop in MTLD signals repetitive wording.
  • Human check: Three annotators ranked 10 BioMistral runs on English 6A20, with low agreement (Cohen's κ ≈ 0).
    • The best runs (8, 9) had moderate MTLD gains of about 27–37.
    • The worst runs had extreme gains: 82 for run 1 and 8 for run 5.
  • Multilingual results: PRISM produced no German or French paraphrases. Code is released at github.com/uit-hdl/multilingual-text-simplification-icd-11.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing · Life sciences & medicine

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bionlp-1.36

@inproceedings{pelloni2026evaluation,
  title = {{Evaluation of Multilingual Text Simplification for the Mental Health Domain: Exploring Small Language Models}},
  author = {Pelloni, Olga and Just, Sandra and Bongo, Lars},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.36},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.36}
}

References

41 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ankit Aich, Tingting Liu, Salvatore Giorgi, Kelsey Isman, Ruhshana Bobojonova, Lyle Ungar, and Brenda Curtis. 2025. Language models in digital psychiatry: challenges with simplification of healthcare materials. NPP—Digital Psychiatry and Neuroscience, 3(1):10. ↗
  2. Jonathan Anderson. 1983. Lix and rix: Variations on a little-known readability index. Journal of Reading, 26(6):490–496. find
  3. Shivam Bansal and Chaitanya Aggarwal. 2026. Textstat: Python readability statistics library. https://github.com/textstat/textstat. ↗
  4. Chandrayee Basu, Rosni Vasu, Michihiro Yasunaga, and Qian Yang. 2023. Med-easi: Finely annotated dataset and models for controllable simplification of medical texts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14093–14101. ↗
  5. Carl-Hugo Björnsson. 1968. Läsbarhet: hur skall man som författare nå fram till läsarna? Bokförlaget Liber. find
  6. Ioana Buhnila. 2025. Explain this medical term in my language: A case study of small language models for medical paraphrase generation. In 3rd UniDive Workshop, pages 1–4, Budapest, Hungary. find
  7. Ioana Buhnila, Aman Sinha, and Mathieu Constant. 2024. Retrieve, generate, evaluate: A case study for medical paraphrases generation with small language models. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 189–203, Bangkok, Thailand. Association for Computational Linguistics. find
  8. Ioana Buhnila and Amalia Todirascu. 2024. LARGEMED: A resource for identifying and generating paraphrases for French medical terms. In Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024, pages 141–151, Torino, Italia. ELRA and ICCL. find
  9. Jeanne Sternlicht Chall and Edgar Dale. 1995. Readability revisited: The new Dale-Chall readability formula. Brookline Books. find
  10. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283. ↗
  11. Michael A Covington and Joe D McFall. 2010. Cutting the Gordian knot: The moving-average type–token ratio (MATTR). Journal of quantitative linguistics, 17(2):94–100. ↗
  12. Mohamed Elgaar and Hadi Amiri. 2025. LingConv: An interactive toolkit for controlled paraphrase generation with linguistic attribute control. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 42–51, Suzhou, China. Association for Computational Linguistics. ↗
  13. Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221. ↗
  14. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. ↗
  15. Agasthya Gangavarapu. 2024. Introducing L2M3, a multilingual medical large language model to advance health equity in low-resource regions. arXiv preprint arXiv:2404.08705. ↗
  16. Sian Gooding. 2022. On the ethical considerations of text simplification. In Ninth Workshop on Speech and Language Processing for Assistive Technologies (SLPAT-2022), pages 50–57, Dublin, Ireland. Association for Computational Linguistics. ↗
  17. Ernest Gralton, Marilyn Sher, and Coro Drew Lopez. 2010. Information and readability issues for psychiatric patients: e-learning for users. The Psychiatrist, 34(9):376–380. ↗
  18. Łukasz Grzybowski, Jakub Pokrywka, Michał Ciesiółka, Jeremi Ignacy Kaczmarek, and Marek Kubis. 2025. Polish-English medical knowledge transfer: A new benchmark and results. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 9042–9063, Suzhou, China. Association for Computational Linguistics. find
  19. R. Gunning. 1952. The Technique of Clear Writing. McGraw-Hill. find
  20. James E Harrison, Stefanie Weber, Robert Jakob, and Christopher G Chute. 2021. ICD-11: an international classification of diseases for the twenty-first century. BMC medical informatics and decision making, 21(Suppl 6):206. ↗
  21. Sadid A Hasan and Oladimeji Farri. 2019. Clinical natural language processing with deep learning. In Data Science for Healthcare: Methodologies and Applications, pages 147–171. Springer. ↗
  22. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. Preprint, arXiv:2310.06825. ↗
  23. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease formula) for Navy enlisted personnel. Technical report, Naval technical training command. ↗
  24. Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. BioMistral: A collection of open-source pretrained large language models for medical domains. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5848–5864, Bangkok, Thailand. Association for Computational Linguistics. find our card
  25. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗ our card
  26. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  27. Camilla Lindholm and Ulla Vanhatalo. 2021. Handbook of easy languages in Europe. Frank & Timme. ↗
  28. G Harry Mc Laughlin. 1969. SMOG grading – a new readability formula. Journal of reading, 12(8):639– 646. find
  29. Philip M McCarthy and Scott Jarvis. 2010. MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2):381–392. ↗
  30. Adam P Natoli. 2026. Leveraging artificial intelligence to linguistically compare test translations: A methodological introduction and demonstration. Assessment, 33(2):163–177. ↗
  31. Misako Nomura, Gyda Skat Nielsen, and Bror Tron-backe. 2010. Guidelines for easy-to-read materials. IFLA professional reports, (120):0_1. Pennsylvania, USA. Association for Computational Linguistics. find
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. ↗
  33. Wulf Rössler. 2016. The stigma of mental disorders: A millennia-long history of social exclusion and prejudices. The EMBO Reports, 17(9):1250–1253. find
  34. Karen Scholz and Markus Wenzel. 2025. Evaluating readability metrics for German medical text simplification. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6049– 6062, Abu Dhabi, UAE. Association for Computational Linguistics. find
  35. Lucas Shen. 2022. LexicalRichness: A small module to compute textual lexical richness. https://github.com/LSYS/lexicalrichness. DOI: 10.5281/zenodo.6607007. ↗
  36. Brian Thompson and Matt Post. 2020. Paraphrase generation as zero-shot multilingual translation: Disentangling semantic similarity from lexical and syntactic diversity. In Proceedings of the Fifth Conference on Machine Translation (Volume 1: Research Papers), Online. Association for Computational Linguistics. find our card
  37. Rui Yang, Qingcheng Zeng, Keen You, Yujie Qiao, Lucas Huang, Chia-Chun Hsieh, Benjamin Rosand, Jeremy Goldwasser, Amisha Dave, Tiarnan Keenan, and 1 others. 2024. Ascle—a python natural language processing toolkit for medical text generation: development and evaluation study. Journal of medical Internet research, 26:e60601. find
  38. Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. 2022. BioBART: Pretraining and evaluation of a biomedical generative language model. In Proceedings of the 21st Workshop on Biomedical Language Processing, pages 97–109. find
  39. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International conference on machine learning, pages 11328–11339. PMLR. ↗ our card
  40. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675. ↗
  41. A Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pelloni-2026-evaluation-multilingual-text-simplification-mental/