Elamur Research⌕
☰
home / papers / 2026 / Evaluation of Document-Level Text Simplification in Japanese

Evaluation of Document-Level Text Simplification in Japanese

Summary

Reliable automatic metrics for judging whole-document simplification are scarce. For Japanese, there were none, and the older D-SARI metric correlates only weakly with human ratings. The authors address this by building the first human-annotated benchmark for Japanese document-level simplification. They took Wikipedia articles from the JADOS corpus and paired each with six simplified versions: human references, a fine-tuned BART, two open LLMs and GPT-4o. Native speakers rated each pair using guidelines that cover two meaning-related criteria (necessity and sufficiency) and two simplicity criteria (sentence-level and document-level). The team then tested several commercial LLMs as judges under different prompting setups and compared their scores with the human ratings. GPT-4o and Gemini 2.5 Pro matched the annotators well even without examples, while Claude 4 Sonnet lagged on the simplicity criteria. The models also showed a systematic bias: they tended to rate documents as less simple than humans did, especially texts with many kanji or with few but long sentences. The work shows that LLM judges can make Japanese simplification evaluation scalable, but their language-specific blind spots need attention.

Key points

  • Dataset size and sources: 1,128 source–target pairs, built from 188 articles in the JADOS Wikipedia validation set. The original set has 195 "Good" or "Featured" articles; texts over 2,000 characters were dropped.
  • Six output types per article:
    • JADOS human references
    • bart-large-japanese
    • Llama-3.1-Swallow-8B-Instruct-v0.2
    • gemma-2-9b-it
    • GPT-4o-2024-11-20 in 0-shot mode
    • GPT-4o-2024-11-20 in 1-shot mode
  • Output length: human references average about 148 characters. LLM outputs ran longer even when asked for roughly 150.
  • Judge setups: GPT-4o, Claude 4 Sonnet and Gemini 2.5 Pro were tested in 0-shot, 1-shot and few-shot modes, each with baseline, majority-vote, reasoning and English-prompt variants. The bias analysis also includes ModernBERT-ja and BERT-Base-ja.
  • Bias and availability: underestimation of document-level simplicity appeared across most configurations, and the authors probed the kanji effect by rewriting kanji as hiragana. The dataset is released under CC BY 4.0 (GitHub: SDS-NLP/JADOS-eval).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/23krjsuxizo3

@inproceedings{yamashita2026evaluation,
  title = {{Evaluation of Document-Level Text Simplification in Japanese}},
  author = {Yamashita, Iori and Tanaka, Hikari and Kiyama, Hajime and Bian, Kexin and Chen, Zhousi and Komachi, Mamoru},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/23krjsuxizo3},
  url = {https://doi.org/10.63317/23krjsuxizo3}
}

References

21 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2019. Cross-Sentence Transformations in Text Simplification. In Proceedings of the 2019 Workshop on Widening NLP, pages 181–184, Florence, Italy. Association for Computational Linguistics. find
  2. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. LLMs instead of Human Judges? a Large Scale Empirical Study across 20 NLP Evaluation Tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 238–255, Vienna, Austria. Association for Computational Linguistics. find our card
  3. Sofia Blinova, Xinyu Zhou, Martin Jaggi, Carsten Eickhoff, and Seyed Ali Bahrainian. 2023. SIM-SUM: Document-level Text Simplification via Simultaneous Summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9927–9944, Toronto, Canada. Association for Computational Linguistics. find
  4. Liam Cripwell, Joël Legrand, and Claire Gardent. 2024. Evaluating Document Simplification: On the Importance of Separately Assessing simplicity and meaning preservation. In Proceedings of the 3rd Workshop on Tools and Resources for People with REAding DIfficulties (READI) @ LREC-COLING 2024, pages 1–14, Torino, Italia. ELRA and ICCL. ↗
  5. Xiyan Fu and Wei Liu. 2025. How Reliable is Multilingual LLM-as-a-Judge? ArXiv preprint arXiv:2505.12201. ↗
  6. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A Survey on LLM-as-a-Judge. ArXiv preprint arXiv:2411.15594. ↗
  7. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. ArXiv preprint arXiv:2412.05579. ↗ our card
  8. Junyu Lu, Kai Ma, Kaichun Wang, Kelaiti Xiao, Roy Ka-Wei Lee, Bo Xu, Liang Yang, and Hongfei Lin. 2025. Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement. In Findings of the Association for Computational Linguistics: ACL 2025, pages 5609–5626, Vienna, Austria. Association for Computational Linguistics. find
  9. Itai Mondshine, Tzuf Paz-Argaman, and Reut Tsarfaty. 2025. Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1331–1354, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  10. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗ our card
  11. Matthew Shardlow. 2014. A Survey of Automated Text Simplification. International Journal of Advanced Computer Science and Applications (IJACSA) , Special Issue on Natural Language Processing 2014, 4(1). ↗
  12. Laura Vásquez-Rodríguez, Matthew Shardlow, Piotr Przybyła, and Sophia Ananiadou. 2023. Document-level Text Simplification with Coherence Evaluation. In Proceedings of the Second Workshop on Text Simplification, Accessibility and Readability, pages 85–101, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
  13. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations. ↗ our card
  14. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc. find our card
  15. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing Statistical Machine Translation for Text Simplification. Transactions of the Association for Computational Linguistics, 4:401–415. 7. ↗ our card
  16. Language Resource References Philippe Laban, Jesse Vig, Wojciech Kryscinski, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2023. SWiPE: A Dataset for Document-level Simplification of Wikipedia Pages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10674–10695, Toronto, Canada. Association for Computational Linguistics. ↗
  17. Rina Miyata, Hinata Koretaka, Hiroki Yamauchi, Daiki Yanamoto, Tomoyuki Kajiwara, Takashi Ninomiya, and Yasuhiro Nishiwaki. 2024. MATCHA: Parallel Corpus for Japanese Text Simplification Based on Professionally Simplified Articles. Journal of Natural Language Processing, 31(2):590–609. ↗
  18. Yoshinari Nagai, Teruaki Oka, and Mamoru Komachi. 2024. A Document-Level Text Simplification Dataset for Japanese. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 459–476, Torino, Italia. ELRA and ICCL. find
  19. Regina Stodden, Omar Momen, and Laura Kallmeyer. 2023. DEplain: A German Parallel Corpus with Intralingual Translations into Plain Language for Sentence and Document Simplification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 16441 – 16463. Association for Computational Linguistics. find
  20. Renliang Sun, Hanqi Jin, and Xiaojun Wan. 2021. Document-Level Text Simplification: Dataset, Criteria and Baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7997–8013, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  21. Toru Urakawa, Yuya Taguchi, Takuro Niitsuma, and Hideaki Tamori. 2024. A Japanese News Simplification Corpus with Faithfulness. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 659–665, Torino, Italia. ELRA and ICCL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yamashita-2026-evaluation-document-level-text-simplification/