名詞役割入れ替えに頑健な日本語含意関係認識

Summary

The paper addresses Japanese natural language inference (NLI) in a hard case: two sentences share the same vocabulary, but their nouns have swapped roles, for example subject and object. This changes the meaning of the sentence. BERT-based models trained on existing corpora almost never recognize such pairs as contradictions. The authors automatically built "shuffled" pairs from sentences in Japanese Wikipedia and the Chunichi Shimbun newspaper archive. The original sentence is the premise, the sentence with swapped nouns is the hypothesis, and the label is "contradiction". To keep the model from guessing the answer from unnatural text, they filtered the pairs using morphological subcategories and the Kyoto University Case Frames dictionary. They then added these data to JSNLI or JSICK and fine-tuned a Japanese BERT. Accuracy on the shuffled data rose from almost zero to 92–99%. Accuracy on the standard test sets stayed about the same. The model therefore distinguishes the meaning of a sentence from a mere bag of words. For Japanese, with its flexible word order, this is essential.

Key points

• How the data were generated. Morphological analysis was done with JUMAN++ and syntactic analysis with KNP (pyknp 0.6.1). Only sentences of up to 40 characters were used, and in each one the first two nouns with the same part-of-speech subcategory were swapped. Only pairs in which every predicate–case combination appears in Kyoto University Case Frames Ver 2.0 were kept.
• Manual check. The authors manually reviewed 100 examples from each source. The meaning actually changed in 81% (Wikipedia) and 84% (newspaper) of cases. It did not change in 12% and 11%. Unnatural sentences made up 7% and 5%.
• Model and training. They used cl-tohoku/bert-base-japanese-whole-word-masking with Hugging Face BertForSequenceClassification. Parameters: AdamW, 10 epochs, lr 2e-5, batch 16. In the JSNLI training data, about 20k pairs labeled contradiction were replaced with roughly the same number of shuffled Wikipedia pairs. About 300 such pairs were added to JSICK.
• Results. Models trained only on JSNLI or JSICK reached 0.1–4.4% accuracy on the shuffled test sets (Wikipedia about 14k pairs, newspaper 10,599 pairs): they labeled almost all pairs as "entailment". With the added data, accuracy reached 99.69/96.85 (JSNLI+) and 98.96/92.34 (JSICK+). On the standard test sets: JSNLI 91.93 → 91.34, JSICK 86.44 → 87.94.
• Limitations. The model most often errs on swaps around the particle "の", because there the meaning sometimes does not change. In future work, the authors want to handle swaps that preserve meaning and to test the method on longer sentences.