Elamur ResearchβŒ•
☰
home / papers / 2026 / Assessing Small Language Models as Text Simplification Evaluators

Assessing Small Language Models as Text Simplification Evaluators

Summary

Automatic evaluation of text simplification usually depends on learned metrics such as LENS and its reference-free variant LENS-SALSA. These metrics need annotated training data and are expensive to build. The authors ask whether small open-weight instruction-tuned LLMs, prompted as judges, could replace them.

They tested Phi-3-mini-4k-Instruct, Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct in two settings:

  • Reference-based: compared against LENS.
  • Reference-free: compared against LENS-SALSA.

Each judge gave a single 0–100 score covering meaning preservation, fluency and simplicity. The scores were correlated with human ratings on SimpEval 2022, Wiki-DA and Newsela-LIKERT.

The LLM judges reached moderate correlations. In some cases they came close to the dedicated metrics, especially when references were available. Overall, LENS and LENS-SALSA were the more consistent and generally stronger evaluators. The smallest model was unreliable without references.

The study helps decide when a cheap, flexible LLM judge is good enough and when a specialized metric is still worth using. It covers English sentence-level simplification only.

Key points

  • Models and baselines: three judges of about 3.8B–8B parameters were compared with LENS, LENS-SALSA (HuggingFace checkpoints, top-k with k=3), FKGL, BLEU, SARI and BERTScore.
  • Correlation measures: Pearson for direct-assessment data (Wiki-DA, Newsela-LIKERT) and a Kendall Ο„-like correlation for the SimpEval 2022 rankings.
  • Reference-based setting: LENS was strongest overall, for example 0.807 on Wiki-DA fluency and 0.621 on Newsela fluency. The LLM judges scored roughly 0.6–0.76 on Wiki-DA dimensions.
  • Reference-free setting:
    • LENS-SALSA reached 0.701, 0.676 and 0.640 on Wiki-DA (fluency, meaning, simplicity).
    • Qwen2.5-7B-Instruct exceeded it on some meaning scores (about 0.80 on Wiki-DA and 0.57 on Newsela, as read from the extracted table).
    • Phi-3-mini produced near-zero correlations.
    • The LLM results on SimpEval were erratic.
  • Limitations and future work: the study is English-only and sentence-level, with no large proprietary models. Proposed next steps include better prompting, calibration, pairwise preference prompting, and hybrid LLM-plus-learned-metric approaches.

Our summary of the paper, in our own words β€” not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation Β· Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/54qbtfwgkz9j

@inproceedings{navarrete2026assessing,
  title = {{Assessing Small Language Models as Text Simplification Evaluators}},
  author = {Navarrete, David Carranza and Bakker, Jan and Kamps, Jaap},
  booktitle = {Proceedings of the 2nd Workshop on Evaluating Text Difficulty in a Multilingual Context (DeTermIt! 2026)},
  year = {2026},
  doi = {10.63317/54qbtfwgkz9j},
  url = {https://doi.org/10.63317/54qbtfwgkz9j}
}

References

9 references, 5 of them with a card of ours. β†— leads to the work, Β«findΒ» searches for one we have no id for yet.

  1. Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. The (un)suitability of automatic evaluation metrics for text simplification. Computational Linguistics, 47(4):861–889. β†— our card
  2. Mingqi Gao, Xinyu Hu, Xunjian Yin, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2025. LLM-based NLG evaluation: Current status and challenges. Computational Linguistics, 51:661–687. β†—
  3. David Heineman, Yao Dou, Mounica Maddela, and Wei Xu. 2023. Dancing between success and failure: Edit-level simplification evaluation using SALSA. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3466–3495, Singapore. Association for Computational Linguistics. find
  4. Peter Kincaid, Robert P. Fishburne, Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Research Branch Report 8-75. β†—
  5. Mounica Maddela, Fernando Alva-Manchego, and Wei Xu. 2021. Controllable text simplification with explicit paraphrasing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3536–3553, Online. Association for Computational Linguistics. find our card
  6. Mounica Maddela, Yao Dou, David Heineman, and Wei Xu. 2023. LENS: A learnable evaluation metric for text simplification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16383–16408, Toronto, Canada. Association for Computational Linguistics. find our card
  7. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. find our card
  8. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics (TACL), 4:401–415. β†— our card
  9. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. β†—

This page in Markdown β†’ https://md.research.elamur.ai/free/papers/navarrete-2026-assessing-small-language-models-as/