Elamur Research⌕
☰
home / papers / 2020 / OCNLI: Original Chinese Natural Language Inference

OCNLI: Original Chinese Natural Language Inference

Paper · 2020 · Hai Hu, Kyle Richardson, Liang Xu et al.

Summary

Most progress in natural language inference (NLI) has happened in English. Resources for other languages have mostly been built by machine-translating English corpora, which brings in translation errors, unnatural "translationese" and Western cultural content. OCNLI addresses this gap for Chinese. It is a corpus of about 56,000 premise–hypothesis pairs built entirely from original Chinese text. Premises come from five genres covering both written and spoken language. Hypotheses were written and labelled by native-speaker university students with training in linguistics or language studies. The workflow is modelled on the MNLI protocol but adds new elicitation schemes meant to produce more varied hypotheses. Baselines include standard NLI models and Chinese pre-trained transformers. The strongest, RoBERTa, scores roughly 78% on the test set, about 12 points below human accuracy of roughly 90%. Models trained on the much larger translated XNLI do worse on OCNLI than models trained on OCNLI itself. This shows the value of natively authored data and makes OCNLI a demanding benchmark for Chinese NLU. It is hosted within CLUE.

Key points

  • Data and sources. About 56,000 pairs labelled Entailment, Neutral or Contradiction. Premises come from five genres: GOV, NEWS, LIT, TV and PHONE. Data and code are released at github.com/CLUEbenchmark/OCNLI, with a leaderboard on the CLUE benchmark.
  • Annotation quality. Annotators agreed on labels about 98% of the time under a 3-vote consensus.
  • Elicitation design. Four modes were used: SINGLE, MULTI, MULTIENCOURAGE and MULTICONSTRAINT. Writing several hypotheses per premise yields easy, medium and hard levels. Models fine-tuned on XNLI lose accuracy steadily from SINGLE to MULTICONSTRAINT and mostly from easy to hard.
  • Native vs. translated data. XNLI is about 8 times larger than OCNLI, yet training on it gives lower accuracy on OCNLI. Combining XNLI with OCNLI still underperforms OCNLI alone. On the CLUE diagnostic set, roughly 25,000 OCNLI examples match a model trained on all of XNLI.
  • Annotation artefacts. Bias was checked with PMI. Words such as 任何 ("any") and 没有 ("no") are strongly associated with Contradiction, and 至少 ("at least") with Entailment. Systematic debiasing is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations🎓Indiana University BloomingtonBloomington · United States18 works · 2,320 citations🎓Indiana UniversityBloomington · United States24 works · 2,995 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Reasoning · 510 works

Cite

Cite as Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kuebler and Lawrence S. Moss. 2020. OCNLI: Original Chinese Natural Language Inference. arXiv:2010.05444.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2010.05444 · arXiv 2010.05444

@misc{hu2020ocnli,
  title = {{OCNLI: Original Chinese Natural Language Inference}},
  author = {Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kuebler and Lawrence S. Moss},
  year = {2020},
  doi = {10.48550/arxiv.2010.05444},
  eprint = {2010.05444},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2010.05444}
}

References

60 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lasha Abzianidze. 2015. A Tableau Prover for Natural Logic and Language. In Proceedings of EMNLP, pages 2492–2502. find
  2. Željko Agić and Natalie Schluter. 2017. Baselines and test data for cross-lingual inference. Proceedings of LREC. find
  3. Hossein Amirkhani, Mohammad Azari Jafari, Azadeh Amirak, Zohreh Pourjafari, Soroush Faridan Jahromi, and Zeinab Kouhkan. 2020. Farstail: A Persian Natural Language Inference Dataset. arXiv preprint arXiv:2009.08820. — [arXiv:2009.08820](https://arxiv.org/abs/2009.08820) find
  4. Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. 2019. Don’t take the premise for granted: Mitigating artifacts in natural language inference. Proceedings of ACL. our card
  5. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP. find
  6. Samuel R Bowman, Jennimaria Palomaki, Livio Baldini Soares, and Emily Pitler. 2020. Collecting entailment data for pretraining: New protocols and negative results. arXiv preprint arXiv:2004.11997. — [arXiv:2004.11997](https://arxiv.org/abs/2004.11997) find
  7. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for Natural Language Inference. In Proceedings of ACL. find
  8. Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases. Proceedings of EMNLP. find
  9. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of EMNLP. our card
  10. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP. find
  11. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Josef Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. Technical report, Technical Report LRE 62-051 D-16, The FraCaS Consortium. find
  12. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-Training with Whole Word Masking for Chinese BERT. arXiv preprint arXiv:1906.08101. our card
  13. Ido Dagan, Oren Glickman, Alfio Gliozzo, Efrat Mar-morshtein, and Carlo Strapparava. 2006. Direct word sense matching for lexical substitution. In Proceedings of ACL, pages 449–456. find
  14. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL Recognizing Textual Entailment Challenge. In Proceedings of the PASCAL Challenges Workshop on Recognizing Textual Entailment. find
  15. Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J Gershman, and Noah D Goodman. 2018. Evaluating compositionality in sentence embeddings. In Proceedings of CogSci, pages 1596–1601. find
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL. our card
  17. Shi Feng, Eric Wallace, and Jordan Boyd-Graber. 2019. Misleading Failures of Partial-input Baselines. Proceedings of ACL. find
  18. E Fonseca, L Santos, Marcelo Criscuolo, and S Aluisio. 2016. Assin: Avaliacao de similaridade semantica e inferencia textual. In Computational Processing of the Portuguese Language-12th International Conference, pages 13–15. find
  19. Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Modular Representation Underlies Systematic Generalization in Neural Natural Language Inference Models. arXiv preprint arXiv:2004.14623. — [arXiv:2004.14623](https://arxiv.org/abs/2004.14623) find
  20. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets. In Proceedings of EMNLP-IJCNLP. find
  21. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of NAACL. find
  22. Yuta Hayashibe. 2020. Japanese realistic textual entailment corpus. In Proceedings of LREC. find
  23. Hai Hu, Qi Chen, Kyle Richardson, Atreyee Mukherjee, Lawrence S Moss, and Sandra Kuebler. 2020. MonaLog: a Lightweight System for Natural Language Inference Based on Monotonicity. In Proceedings of SCiL. find
  24. Hai Hu and Sandra Kübler. 2020. Investigating translated Chinese and its variants using machine learning. Natural Language Engineering (Special Issue on NLP for Similar Languages, Varieties and Dialects), pages 1–34. find
  25. Hai Hu, Wen Li, and Sandra Kübler. 2018. Detecting syntactic features of translated Chinese. In Proceedings of the 2nd Workshop on Stylistic Variation, pages 20–28. find
  26. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are Natural Language Inference Models IMPPRESsive? Learning IM-Plicature and PRESupposition. arXiv preprint arXiv:2004.03066. our card
  27. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTaiL: A Textual Entailment Dataset from Science Question Answering. In AAAI. find
  28. Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of ACL. our card
  29. Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial Filters of Dataset Biases. Proceedings of ICLR. find
  30. Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. 2018. Analogical reasoning on Chinese morphological and semantic relations. In Proceedings of ACL. find
  31. Chien-Jer Charles Lin. 2011. Chinese and English relative clauses: Processing constraints and typological consequences. In Proceedings of the 23rd North American Conference on Chinese Linguistics (NACCL-23), pages 191–199. find
  32. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. find
  33. Bill MacCartney. 2009. Natural Language Inference. Ph.D. thesis, Stanford University. find
  34. M. Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, and R. Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of LREC. our card
  35. Pascual Martı́nez-Gómez, Koji Mineshima, Yusuke Miyao, and Daisuke Bekki. 2017. On-demand injection of lexical knowledge for recognising textual entailment. In Proceedings of EACL. our card
  36. R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. In Proceedings of ACL. our card
  37. Anthony McEnery and Zhonghua Xiao. 2004. The Lancaster Corpus of Mandarin Chinese: A corpus for monolingual and contrastive language study. In LREC, pages 1175–1178. find
  38. Yashar Mehdad, Matteo Negri, and Marcello Federico. 2011. Using Bilingual Parallel Corpora for Cross-lingual Textual Entailment. In Proceedings of ACL. find
  39. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress Test Evaluation for Natural Language Inference. In Proceedings of COLING. find
  40. Nikita Nangia and Samuel Bowman. 2019. Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark. In Proceedings of ACL. find
  41. Matteo Negri, Luisa Bentivogli, Yashar Mehdad, Danilo Giampiccolo, and Alessandro Marchetti. 2011. Divide and Conquer: Crowdsourcing the Creation of Cross-lingual Textual Entailment Corpora. In Proceedings of EMNLP. find
  42. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A New Benchmark for Natural Language Understanding. In Proceedings of ACL. find
  43. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics, 7:677–694. our card
  44. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. Proceedings of NAACL. our card
  45. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis Only Baselines in Natural Language Inference. In Proceedings of *SEM. find
  46. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv e-prints arXiv:1910.10683. our card
  47. Livy Real, Erick Fonseca, and Hugo Gonçalo Oliveira. 2020. Organizing the ASSIN 2 shared task. In Proceedings of the ASSIN 2 Shared Task: Evaluating Semantic Textual Similarity and Textual Entailment in Portuguese. find
  48. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of EMNLP. our card
  49. Kyle Richardson, Hai Hu, Lawrence S Moss, and Ashish Sabharwal. 2020. Probing Natural Language Inference Models through Semantic Fragments. In Proceedings of AAAI. our card
  50. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WINOGRANDE: An adversarial Winograd schema challenge at scale. Proceedings of AAAI. find
  51. Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Arivazhagan, Jason Riesa, Ankur Bapna, Orhan Firat, and Karthik Raman. 2020. Evaluating the cross-lingual effectiveness of massively multilingual neural machine translation. Proceedings of AAAI. find
  52. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. Proceedings of ICLR. find
  53. Harsh Trivedi, Heeyoung Kwon, Tushar Khot, Ashish Sabharwal, and Niranjan Balasubramanian. 2019. Repurposing entailment for multi-hop question answering Tasks. Proceedings of NAACL. find
  54. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-purpose Language Understanding Systems. In Advances in Neural Information Processing Systems. our card
  55. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of BlackboxNLP. our card
  56. Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, et al. 2019. Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs. Proceedings of EMNLP. our card
  57. Barbara Wheatley. 1996. CALLHOME Mandarin Chinese Transcripts LDC96T16. find
  58. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of NAACL. find
  59. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipe… our card
  60. Hitomi Yanaka, Koji Mineshima, Pascual Martı́nez-Gómez, and Daisuke Bekki. 2018. Acquisition of Phrase Correspondences using Natural Deduction Proofs. In Proceedings of NAACL. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hu-2020-ocnli-original-chinese-natural/