Elamur Research⌕
☰
home / papers / 2022 / JGLUE: Japanese General Language Understanding Evaluation

JGLUE: Japanese General Language Understanding Evaluation

Summary

Japanese had no GLUE-style benchmark for general natural language understanding. The existing Japanese datasets, such as JSNLI and JSICK, were machine- or human-translated from English, which brings unnatural phrasing and cultural mismatch. The authors built JGLUE natively in Japanese, without translation, using Yahoo! Crowdsourcing. It covers three task families:

  • Text classification: MARC-ja, a binary sentiment set derived from Amazon reviews. JCoLA is planned and will come from another group.
  • Sentence pair classification: JSTS for similarity and JNLI for inference. Both are built from YJ Captions, Japanese captions written for MS COCO images.
  • QA: JSQuAD and JCommonsenseQA.

The construction design uses several devices:

  • Evaluation splits are cleaned through crowd agreement.
  • Low-similarity pairs come from captions of different images.
  • Contradictions are written by workers.
  • Annotations with high disagreement are removed.

The main contribution is the resource itself, a publicly released, multi-task Japanese benchmark that parallels CLUE, FLUE and KLUE. The excerpt provided here omits the model-evaluation section, so baseline scores are not reported in this summary. JGLUE matters because findings from English do not necessarily transfer to Japanese, given its mixed scripts, lack of word spacing and flexible word order.

Key points

  • Split sizes (train / dev / test):
    • MARC-ja: 187,528 / 5,654 / 5,639
    • JSTS: 12,451 / 1,457 / 1,589
    • JNLI: 20,073 / 2,434 / 2,508
    • JSQuAD: 63,870 / 4,475 / 4,470
    • JCommonsenseQA: 9,012 / 1,119 / 1,118
  • MARC-ja construction:
    • 1–2 star reviews become "negative" and 4–5 star reviews become "positive"; 3-star reviews are discarded.
    • About 12,000 reviews were re-labeled by 10 crowd workers each, and only reviews where at least 7 workers agreed were kept for dev/test.
    • The metric is accuracy.
  • JSTS-A: 16,000 same-image caption pairs were rated by 5 workers each, with the mean score used as the gold value. Pairs with rating variance of 1.0 or more were dropped, leaving 10,236.
  • JSTS-B: 4,000 different-image caption pairs were rated the same way, leaving 2,970 pairs that supply low-similarity examples.
  • JNLI labels: JNLI uses SNLI-style labels (entailment, neutral, contradiction). Its sentence pairs largely overlap with JSTS, similar to how SICK and JSICK pair both annotation types.

Interesting ideas

  • I1. Native construction instead of translation: Collecting data directly in the target language avoids translation artifacts and culturally foreign content. This is directly relevant to building a Ukrainian NLU benchmark rather than translating GLUE.
  • I2. Clean only the evaluation splits: Agreement-based crowd verification is spent on dev/test, while the large noisy training set is used as is. This concentrates annotation budget where label quality affects measured scores most.
  • I3. Use image captions as a source of sentence pairs: Captions of the same image yield naturally similar pairs, and captions of different images yield dissimilar ones. This is a cheap way to cover the full similarity range without relying on workers to invent unrelated sentences.
  • I4. Collect rare labels by authoring, not mining: Contradictions seldom occur among naturally sampled pairs, so workers were asked to write them. This fixes label imbalance where mining alone fails.
  • I5. Filter by annotator disagreement, and annotate shared pairs twice: Dropping items with high rating variance, or with weak majority agreement, is a simple quality filter. Giving the same pairs both similarity and inference labels lets the two phenomena be studied together.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

69 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kurihara2022jglue,
  title = {{JGLUE: Japanese General Language Understanding Evaluation}},
  author = {Kurihara, Kentaro and Kawahara, Daisuke and Shibata, Tomohide},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.5715/jnlp.29.711},
  url = {https://doi.org/10.5715/jnlp.29.711}
}

References

39 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Mageed, M., Elmadany, A. A., and Nagoudi, E. M. B. (2021). ARBERT & MARBERT: deep bidirectional transformers for arabic. CoRR, abs/2101.01785. find our card
  2. Bragg, J., Cohan, A., Lo, K., and Beltagy, I. (2021). FLEX: Unifying evaluation for few-shot NLP. In A. Beygelzimer, et al., editors, Advances in Neural Information Processing Systems. find
  3. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. (2017). Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). find
  4. Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., and Zitnick, C. L. (2015). Microsoft coco captions: Data collection and evaluation server. 12 We used MeCab + IPAdic (https://taku910. github.io/mecab/) for word segmentation. find our card
  5. Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., and Palomaki, J. (2020). TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. find our card
  6. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (… find
  7. Hayashibe, Y. (2020). Japanese realistic textual entailment corpus. In LREC2020, pages 6827–6834, Marseille, France, May. European Language Resources Association. find
  8. Hu, J., Ruder, S., Siddhant, A., Neubig, G., Firat, O., and Johnson, M. (2020). XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. find our card
  9. Kakwani, D., Kunchukuttan, A., Golla, S., N.C., G., Bhattacharyya, A., Khapra, M. M., and Kumar, P. (2020). IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Computational Linguistics: EMNLP… find
  10. Keung, P., Lu, Y., Szarvas, G., and Smith, N. A. (2020). The multilingual Amazon reviews corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4563–4568, Online, November. find
  11. Kudo, T., Yamamoto, K., and Matsumoto, Y. (2004). Applying conditional random fields to Japanese morphological analysis. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 230–237, Barcelona, Spain, July. find our card
  12. Le, H., Vial, L., Frej, J., Segonne, V., Coavoux, M., Lecouteux, B., Allauzen, A., Crabbé, B., Besacier, L., and Schwab, D. (2020). FlauBERT: Unsupervised language model pre-training for French. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2479–2490, Marseille, Fra… find our card
  13. Liang, Y., Duan, N., Gong, Y., Wu, N., Guo, F., Qi, W., Gong, M., Shou, L., Jiang, D., Cao, G., Fan, X., Zhang, R., Agrawal, R., Cui, E., Wei, S., Bharti, T., Qiao, Y., Chen, J.-H., Wu, W., Liu, S., Yang, F., Campos, D., Majumder, R., and Zhou, M. (2020). XGLUE: A new benchmark dataset for cross-lin… find our card
  14. Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, P. (2015). Microsoft coco: Common objects in context. find our card
  15. Liu, D., Yan, Y., Gong, Y., Qi, W., Zhang, H., Jiao, J., Chen, W., Fu, J., Shou, L., Gong, M., Wang, P., Chen, J., Jiang, D., Lv, J., Zhang, R., Wu, W., Zhou, M., and Duan, N. (2021). GLGE: A new general language generation evaluation benchmark. In Findings of the Association for Computational Lingu… find
  16. Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., and Zamparelli, R. (2014). A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223, Reykjavik,… find
  17. Miyazaki, T. and Shimizu, N. (2016). Cross-lingual image caption generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1780–1790. find
  18. Morita, H., Kawahara, D., and Kurohashi, S. (2015). Morphological analysis for unsegmented languages using recurrent neural network language model. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2292–2297, Lisbon, Portugal, September. find
  19. Park, S., Moon, J., Kim, S., Cho, W. I., Han, J. Y., Park, J., Song, C., Kim, J., Song, Y., Oh, T., Lee, J., Oh, J., Lyu, S., Jeong, Y., Lee, I., Seo, S., Lee, D., Kim, H., Lee, M., Jang, S., Do, S., Kim, S., Lim, K., Lee, J., Park, K., Shin, J., Kim, S., Park, L., Oh, A., Ha, J.-W., and Cho, K. (20… find
  20. Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., and Van Durme, B. (2018). Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana, June. Association for Computational L… find
  21. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November. find our card
  22. Rodriguez-Penagos, C., Armentano-Oller, C., Villegas, M., Melero, M., Gonzalez, A., de Gibert Bonet, O., and Pio, C. C. (2021). The catalan language CLUB. CoRR, abs/2112.01894. — [arXiv:2112.01894](https://arxiv.org/abs/2112.01894) find
  23. Ruder, S., Constant, N., Botha, J., Siddhant, A., Firat, O., Fu, J., Liu, P., Hu, J., Garrette, D., Neubig, G., and Johnson, M. (2021). XTREME-R: Towards more challenging and nuanced multilingual evaluation. find our card
  24. Seelawi, H., Tuffaha, I., Gzawi, M., Farhan, W., Talafha, B., Badawi, R., Sober, Z., Al-Dweik, O., Freihat, A. A., and Al-Natsheh, H. (2021). ALUE: Arabic language understanding evaluation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 173–184, Kyiv, Ukraine (Virtual… find
  25. Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August. find
  26. Speer, R., Chin, J., and Havasi, C. (2017). Conceptnet 5.5: An open multilingual graph of general knowledge. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. find
  27. Suzuki, M., Matsuda, K., Okazaki, N., and Inui, K. (2018). Construction of a question answering dataset with answerability by reading. In NLP2018. in Japanese. find
  28. Takahashi, N., Shibata, T., Kawahara, D., and Kurohashi, S. (2019). Machine comprehension improves domain-specific Japanese predicate-argument structure analysis. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 98–104, Hong Kong, China, November. find
  29. Talmor, A., Herzig, J., Lourie, N., and Berant, J. (2019). CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long… find
  30. Tsuchiya, M. (2018). Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). find our card
  31. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belg… find our card
  32. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2019). SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, et al., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associa… find our card
  33. Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., and Li, B. (2021). Adversarial glue: A multi-task benchmark for robustness evaluation of language models. find
  34. Warstadt, A., Singh, A., and Bowman, S. R. (2019). Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  35. Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),… find
  36. Xu, L., Hu, H., Zhang, X., Li, L., Cao, C., Li, Y., Xu, Y., Sun, K., Yu, D., Yu, C., Tian, Y., Dong, Q., Liu, W., Shi, B., Cui, Y., Li, J., Zeng, J., Wang, R., Xie, W., Li, Y., Patterson, Y., Tian, Z., Zhang, Y., Zhou, H., Liu, S., Zhao, Z., Zhao, Q., Yue, C., Zhang, X., Yang, Z., Richardson, K., an… find our card
  37. Yanaka, H. and Mineshima, K. (2021). JSICK: Japanese sentences involving compositional knowledge dataset. In JSAI2021. in Japanese. find
  38. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. (2018). HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium,… find our card
  39. Yoshikoshi, T., Kawahara, D., and Kurohashi, S. (2020). Multilingualization of a natural language inference dataset using machine translation. In The Special Interest Group Technical Reports of IPSJ. in Japanese. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/kurihara-2022-jglue-japanese-general-language-understanding/