Elamur Research⌕
☰
home / papers / 2026 / JSTS-Neg: Japanese Semantic Textual Similarity Dataset for Evaluating Negation Understanding Ability

JSTS-Neg: Japanese Semantic Textual Similarity Dataset for Evaluating Negation Understanding Ability

Summary

Large language models still handle negation poorly, yet few negation-focused benchmarks target semantic textual similarity (STS), and Japanese resources are especially scarce. The earlier Japanese set N-JSTS attached a negation cue at the final position of each sentence in a pair, which produced only clausal negation and offered no way to isolate negation's effect. The authors address this with JSTS-Neg, derived from JSTS (part of JGLUE) following the procedure Yoshida et al. (2025) developed for JNLI-Neg. Rules insert the negative morphemes "ない" or "ず" into verbs, adjectives or adjectival nouns, creating one variant for each possible insertion site. An LLM-based well-formedness check discards ungrammatical or nonsensical outputs. The surviving sentences are paired, with either one side or both negated, and human annotators assign similarity scores. Every item forms a negation minimal pair with its source, so comparing a model's predictions across the pair reveals how much it attends to negation alone. Tests on 20 open LLMs showed that, under certain conditions, models often score sentence pairs as if the negation cue were absent. The dataset and code are public.

Key points

  • About 85% of JSTS-Neg instances contain a negation cue. The set covers both clausal and sub-clausal negation, whereas N-JSTS (Uchida and Nanjo, 2024) is limited to clausal negation.
  • Affixal negation and double negation are deliberately excluded, and the only target cues are "ない" and "ず".
  • Analysis contrasts control (original) and treatment (negated) groups and separates important negation cues, which shift the gold score, from unimportant ones, which leave it unchanged.
  • The evaluated models comprise 20 open instruction-tuned LLMs:
    • Japanese-oriented: llm-jp-3.1, Swallow variants, rinna Youko/Baku.
    • Multilingual: Llama 3/3.1, Mistral/Ministral/Mistral Nemo, Gemma 2/3/3n.
  • Data and source code are available at github.com/reiko-y/JSTS-Neg (LREC 2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4rywvofmkvff

@inproceedings{yuasa2026jsts,
  title = {{JSTS-Neg: Japanese Semantic Textual Similarity Dataset for Evaluating Negation Understanding Ability}},
  author = {Yuasa, Reiko and Kato, Yoshihide and Matsubara, Shigeki},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4rywvofmkvff},
  url = {https://doi.org/10.63317/4rywvofmkvff}
}

References

26 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazuki Fujii, Kensuke Fukumoto, Takuya Fukushima, Namgi Han, Yuto Harada, Chikara Hashimoto, Tatsuya Hiraoka, Shohei Hisada, Sosuke Hosokawa, Lu Jie, Keisuke Kamata, Teruhito Kanazawa, Hiroki Kanezashi, Hiroshi Kat… — [arXiv:2407.03963](https://arxiv.org/abs/2407.03963) find
  2. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–382. find
  3. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual LLM adaptation: Enhancing Japanese language capabilities. In Proceedings of the First Conference on Langua… find our card
  4. Namgi Han, Nobuhiro Ueda, Masatoshi Otake, Satoshi Katsumata, Keisuke Kamata, Hirokazu Kiyomaru, Takashi Kodama, Saku Sugawara, Bowen Chen, Hiroshi Matsuda, Yusuke Miyao, Yugo Murawaki, and Koki Ryu. 2024. llm-jp-eval: Automatic evaluation tool for Japanese large language models. In Proceedings of t… find
  5. Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022. An analysis of negation in natural language understanding corpora. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022) (Short Papers), volume 2. find
  6. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  7. Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu,… find
  8. Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. 2024. Building a large Japanese web corpus for large language models. In Proceedings of the First Conference on Language Modeling (COLM), University of P… find
  9. Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Ni… find
  10. Kei Sawada, Tianyu Zhao, Makoto Shing, Kentaro Mitsui, Akio Kaga, Yukiya Hono, Toshiaki Wakatsuki, and Koh Mitsuda. 2024. Release of pre-trained models for the Japanese language. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation… find
  11. Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. 2023. Language models are not naysayers: an analysis of language models on negation benchmarks. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023). find
  12. Yuxia Wang, Minghan Wang, and Preslav Nakov. 2024. Rethinking STS and NLI in large language models. In Findings of the Association for Computational Linguistics (EACL 2024), pages 965– 982, St. Julian’s, Malta. Association for Computational Linguistics. find
  13. Reiko Yuasa, Asahi Yoshida, Yoshihide Kato, and Shigeki Matsubara. 2025. Evaluation of japanese language understanding benchmarks from the perspective of negation. In Proceedings of the Thirty-first Annual Meeting of the Association for Natural Language Processing. In Japanese. 8. find
  14. Language Resource References Iker García-Ferrero, Begoña Altuna, Javier Alvez, Itziar Gonzalez-Dios, and German Rigau. 2023. This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models. Association for Computational Linguistics (ACL). find
  15. Mareike Hartmann, Miryam de Lhoneux, Daniel Hershcovich, Yova Kementchedjhieva, Lukas Nielsen, Chen Qiu, and Anders Søgaard. 2021. A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs. Association for Computational Linguistics (ACL). find
  16. Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An Analysis of Natural Language Inference Benchmarks through the Lens of Negation. Association for Computational Linguistics (ACL). find our card
  17. Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. JGLUE: Japanese General Language Understanding Evaluation. European Language Resources Association (ELRA). find our card
  18. Suguru Matsuyoshi, Ryo Otsuki, and Fumiyo Fukumoto. 2014. Annotating the Focus of Negation in Japanese Text. European Language Resources Association (ELRA). find
  19. Abhilasha Ravichander, Matt Gardner, and Ana Marasović. 2022. CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation. Association for Computational Linguistics (ACL). find
  20. Taiga Someya and Yohei Oseki. 2023. JBLiMP: Japanese Benchmark of Linguistic Minimal Pairs. Association for Computational Linguistics (ACL). find
  21. Ekaterina Taktasheva, Maxim Bazhukov, Kirill Koncha, Alena Fenogenova, Ekaterina Artemova, and Vladislav Mikhailov. 2024. RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs. Association for Computational Linguistics (ACL). find
  22. Takumi Uchida and Hiroaki Nanjo. 2023. Data Augmentation by Contrapositives for Recognizing Entailment in Sentences with Negation Expressions. In Japanese. find
  23. Takumi Uchida and Hiroaki Nanjo. 2024. Performance verification of natural language understanding in sentences with negation expressions. In Japanese. find
  24. Hitomi Yanaka and Koji Mineshima. 2021. Assessing the Generalization Capacity of Pre-trained Language Models through Japanese Adversarial Natural Language Inference. Association for Computational Linguistics (ACL). find
  25. Hitomi Yanaka and Koji Mineshima. 2022. Compositional Evaluation on Japanese Textual Entailment and Similarity. MIT Press. find
  26. Asahi Yoshida, Yoshihide Kato, Yasuhiro Ogawa, and Shigeki Matsubara. 2025. Construction of a Japanese Language Inference Dataset for Evaluating Negation Understanding Ability. In Japanese. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuasa-2026-jsts-neg-japanese-semantic-textual/