Elamur Research⌕
☰
home / papers / 2022 / Compositional Evaluation on Japanese Textual Entailment and Similarity

Compositional Evaluation on Japanese Textual Entailment and Similarity

Paper · 2022 · Hitomi Yanaka and Koji Mineshima

Summary

Japanese has lacked an NLI/STS benchmark aligned with existing multilingual resources. Its case particles and flexible argument order make it a useful test of whether language models actually process sentence structure. The authors built JSICK by having an expert translator render every English sentence of SICK into Japanese. The translator followed guidelines that preserve phenomena such as quantification, negation, passives and the definite/indefinite distinction. Bilinguals validated the output, and the gold labels were re-annotated. The authors also derived a stress-test set by rewriting JSICK pairs. Scrambling noun-phrase arguments keeps the meaning, while swapping or deleting case particles changes it. In baseline experiments, Japanese RoBERTa-large scored highest and multilingual models came close. Across SICK versions in several languages, multilingual models were weaker on anaphora, disjunction and additive particles. On the stress test, both Japanese and multilingual models mostly repeated their original predictions, which means they largely ignore word order and case marking. A small targeted data augmentation partly corrected this. JSICK thus supports controlled cross-lingual study of compositional inference.

Key points

  • JSICK translates all 6,077 SICK sentences, which make up 9,927 sentence pairs. The translator never saw the entailment labels, and no examples were discarded after validation.
  • Japanese has no articles, so the translation marks indefinite subjects with the particle ga and definite subjects with the topic marker wa.
  • The stress test covers three transformations: case-scrambling, particle-swapping and particle-deleting. Model sensitivity is measured as the share of predictions left unchanged versus the original pairs, and for STS as the Pearson correlation between original and rephrased predictions.
  • The authors added 300 rephrased training examples per transformation type. For jaRoBERTa-large, this lowered the unchanged-prediction rate for particle-swapping from 99.0% to 69.2% and for particle-deleting from 98.3% to 69.1%. Case-scrambling stayed near-invariant (98.9% → 97.7%), which is the desired behavior.
  • The dataset is public at github.com/verypluming/JSICK. It complements earlier Portuguese and Dutch SICK translations to form a parallel multilingual SICK resource.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan90 works · 9,156 citations🎓Keio UniversityTokyo · Japan7 works · 882 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Hitomi Yanaka and Koji Mineshima. 2022. Compositional Evaluation on Japanese Textual Entailment and Similarity. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00518

@article{yanaka2022compositional,
  title = {{Compositional Evaluation on Japanese Textual Entailment and Similarity}},
  author = {Hitomi Yanaka and Koji Mineshima},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00518},
  url = {https://doi.org/10.1162/tacl_a_00518}
}

References

67 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval…
  2. Hossein Amirkhani, Mohammad Azari Jafari, Azadeh Amirak, Zohreh Pourjafari, Soroush Faridan Jahromi, and Zeinab Kouhkan. 2020. FarsTail: A Persian natural language inference dataset. CoRR, cs.CL/2009.08820. Version 1. find
  3. Masayuki Asahara and Yuji Matsumoto. 2003. ipadic version 2.7.0 User’s Manual. Nara Institute of Science and Technology. find
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. https://doi.org/10.18653/v1/D15 -1075 ↗
  5. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14.
  6. Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536–4546. https://doi.org/10 .18653/v1/2020.emnlp-main.3… find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… find
  8. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485. https://doi.… ↗
  9. Robin Cooper, Richard Crouch, Jan van Eijck, Chris Fox, Josef van Genabith, Jan Jaspers, Hans Kamp, Manfred Pinkal, Massimo Poesio, Stephen Pulman. 1994. FraCaS–a framework for computational semantics. Deliverable, D6. find
  10. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190. https://doi.org/10.1007/11736790 9 ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  12. Gottlob Frege. 1963. Compound thoughts. Mind, 72(285):1–17. https://doi.org/10.1093/mind/LXXII.285.1 — [doi:10.1093/mind/lxxii.285.1](https://doi.org/10.1093/mind/lxxii.285.1) ↗
  13. William Gantt, Benjamin Kane, and Aaron Steven White. 2020. Natural language inference with mixed effects. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 81–87.
  14. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655. https://doi.org/10.18653/v1/P18-2103 ↗
  15. Emily Goodwin, Koustuv Sinha, and Timothy J. O’Donnell. 2020. Probing linguistic systematicity. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1958–1969. https://doi.org/10.18653/v1/2020.acl-main.177 ↗
  16. Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar. 2021. BERT & family eat word salad: Experiments with text understanding. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, pages 12946–12954. https://doi.org/10.1609/aaai.v35i14 .17531 — [doi:10.1609/aaai.v35i14](https://doi.org/10.1609/aaai.v35i14) ↗
  17. Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020. KorNLI and KorSTS: New benchmark datasets for Korean natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 422–430. https://doi.org/10.18653/v1/2020.findings -emnl… ↗
  18. John Hawkins. 1978. Definiteness and Indefiniteness. A Study in Reference and Grammaticality Prediction, Routledge. find
  19. Yuta Hayashibe. 2020. Japanese realistic textual entailment corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6827–6834. find
  20. Irene Heim. 1982. The Semantics of Definite and Indefinite Noun Phrases. Ph.D. thesis, UMass Amherst. find
  21. Jack Hessel and Alexandra Schofield. 2021. How effective is BERT without word ordering? implications for language understanding and data privacy. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language… find
  22. John Hinds. 1986. Japanese: Descriptive Grammar. Croom Helm. find
  23. Hajime Hoji. 1985. Logical Form Constraints and Configurational Structures in Japanese. Ph.D. Thesis. University of Washington. find
  24. Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence Moss. 2020. OCNLI: Original Chinese Natural Language Inference. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3512–3526. https://doi.org/10 .18653/v1/2020.findings-emnlp .314
  25. Theo Janssen and Barbara Partee. 1997. Compositionality. Johan van Benthem and Alice ter Meulen, editors, Handbook of Logic and Language, pages 417–473. Elsevier. https://doi.org/10.1016/B978 -044481714-3/50011-4 — [doi:10.1016/b978](https://doi.org/10.1016/b978) ↗
  26. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293. https://doi.org/10 .18653/v1…
  27. Aikaterini-Lida Kalouli, Livy Real, and Valeria de Paiva. 2017. Textual inference: Getting logic from humans. In Proceedings of the 12th International Conference on Computational Semantics (IWCS) — Short papers. find
  28. Jerrold Katz and Jerry Fodor. 1963. The structure of a semantic theory. Language, 39(2):170–210. https://doi.org/10.2307/411200 — [doi:10.2307/411200](https://doi.org/10.2307/411200) ↗
  29. Ai Kawazoe, Ribeka Tanaka, Koji Mineshima, and Daisuke Bekki. 2017. An inference problem set for evaluating semantic theories and semantic processing systems for Japanese. In New Frontiers in Artificial Intelligence, pages 58–65. https://doi.org/10.1007/978-3-319-50953-2_5 — [doi:10.1007/978-3-319-50953-2_5](https://doi.org/10.1007/978-3-319-50953-2_5) ↗
  30. Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… find
  31. Hang Le, Loı̈c Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised language model pre-training for French. In Proceedings of the 12th Language Resources and Evaluation Conferenc…
  32. Roger Levy and Galen Andrew. 2006. Tregex and tsurgeon: Tools for querying and manipulating tree data structures. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06). find
  33. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Mi… — XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
  34. Tal Linzen. 2020. How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5210–5217. https://doi.org/10 .18653/v1/2020.acl-main.465
  35. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. CoRR, cs.CL/1907.11692. Version 1.
  36. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages…
  37. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. https://doi.org/10 .18653/v1/P19-1334
  38. Richard Montague. 1973, The proper treatment of quantification in ordinary English, K. J. J. Hintikka, J. Moravcsic, and P. Suppes, editors, Approaches to Natural Language, pages 221–242. Reidel, Dordrecht. https://doi.org/10 .1007/978-94-010-2506-5_10 find
  39. Hajime Morita, Daisuke Kawahara, and Sadao Kurohashi. 2015. Morphological analysis for unsegmented languages using recurrent neural network language model. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2292–2297. https://doi.org/10 .18653/v1/D15-127… find
  40. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353. find
  41. Kimiko Nakanishi and Satoshi Tomioka. 2004. Japanese plurals are exceptional. Journal of East Asian Linguistics, 13(2):113–140. https://doi.org/10.1023/B:JEAL .0000019058.46668.c1 — [doi:10.1023/b:jeal](https://doi.org/10.1023/b:jeal) ↗
  42. Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Ky… find
  43. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. https://doi.org/10.1162/tacl_a_00293 ↗
  44. Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1145–1160. https://doi.org/10 .18653/v1/2021…
  45. Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lon…
  46. Livy Real, Ana Rodrigues, Andressa Vieira e Silva, Beatriz Albiero, Bruna Thalenberg, Bruno Guide, Cindy Silva, G. Lima, Igor C. S. Câmara, Miloš Stanojević, Rodrigo Souza, and Valeria de Paiva. 2018. SICK-BR: A portuguese corpus for inference. In Computational Processing of the Portuguese Langua… — [doi:10.1007/978-3-319-99722-3_31](https://doi.org/10.1007/978-3-319-99722-3_31) find
  47. Ohad Rozen, Vered Shwartz, Roee Aharoni, and Ido Dagan. 2019. Diversify your datasets: Analyzing generalization via controlled variance in adversarial datasets. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 196–205. https://doi.org/10 .18653/v1/K19-1… find
  48. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? On the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
  49. Mamoru Saito. 1985. Some Asymmetries in Japanese and their Theoretical Implications. Ph.D. thesis, NA Cambridge. find
  50. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. https://doi.org/10 .1109/ICASSP.2012.6289079 find
  51. Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2021. ALUE: Arabic language understanding evaluation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 1…
  52. Tatiana Shavrina, Alena Fenogenova, Emelyanov Anton, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. RussianSuperGLUE: ARussian language understanding evaluation benchmark. In Proceedings of the 2020 Conference on… find
  53. Masayoshi Shibatani. 1990. The Languages of Japan, Cambridge University Press. find
  54. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021a. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. CoRR, cs.CL/2104.06644. Version 1. https://doi.org/10.18653/v1/2021.emnlp -main.230 ↗
  55. Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021b. UnNatural Language Inference. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing…
  56. Arseny Tolmachev, Daisuke Kawahara, and Sadao Kurohashi. 2018. Juman++: A morphological analysis toolkit for scriptio continua. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 54–59. https://doi.org/10.18653/v1/D18-2010 ↗
  57. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations (ICLR). https://doi.org/10.18653/v1/W18-5446 ↗
  58. Jennifer C. White and Ryan Cotterell. 2021. Examining the inductive bias of neural language models with artificial languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… find
  59. Gijs Wijnholds and Michael Moortgat. 2021. SICK-NL: A dataset for Dutch natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1474–1479. https://doi.org/10 .18653/v1/2021.eacl-main.126 find
  60. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
  61. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipe… — CLUE: A Chinese Language Understanding Evaluation Benchmark
  62. Hitomi Yanaka, Koji Mineshima, and Kentaro Inui. 2021. Exploring transitivity in neural NLI models through veridicality. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 920–934. https://doi.org/10.18653/v1 /2021.eacl-…
  63. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin…
  64. Masashi Yoshikawa, Hiroshi Noji, and Yuji Matsumoto. 2017. A* CCG parsing with a supertag and dependency factored model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 277–287. https://doi.org/10.18653/v1/P17 -1026 ↗
  65. Takumi Yoshikoshi, Daisuke Kawahara, and Sadao Kurohashi. 2020. Multilingualization of natural language inference datasets using machine translation (in Japanese). In Proceedings of the 244th Meeting of Natural Language Processing. find
  66. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proceedings of the International Conference on Learning Representations (ICLR). find
  67. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yanaka-2022-compositional-evaluation-japanese-textual-entailment/