Elamur Research⌕
☰
home / papers / 2021 / FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark

FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark

Paper · 2021 · Liang Xu, Xiaojing Lu, Chenyang Yuan et al.

Summary

Chinese NLP had no shared benchmark for comparing few-shot methods, so new techniques could not be measured against each other under common conditions. FewCLUE addresses this gap with nine Chinese NLU tasks: single-sentence classification, sentence-pair classification, and reading-comprehension-style tasks. Six tasks come from CLUE and three are newly added. The training sets are very small. Each task also includes several train/dev splits, a public test set, a private test set for the leaderboard, and up to 20k unlabeled examples for semi-supervised research.

The authors compared five few-shot methods (PET, ADAPET, LM-BFF, P-tuning, EFL) with standard fine-tuning, zero-shot learning, and human performance, using RoBERTa and ERNIE 1.0 as backbones. No method won across all backbones, and every method stayed far below human accuracy.

The benchmark served as the NLPCC 2021 few-shot contest and is released with a toolkit and an online leaderboard.

Key points

  • Training set sizes: tasks with ≤3 labels get 32 examples in total. TNEWS gets 16 per class. CSLDCP (67 labels) and IFLYTEK (119 labels) get 8 per class. CHID gets 42 (6 for each answer position).
  • Overall scores: humans reach 82.5, compared with 44.1 for fine-tuned RoBERTa, 44.6 for zero-shot RoBERTa, and 29.0 for the majority-class baseline.
  • Best method per backbone: in the results table, P-tuning is best with RoBERTa (59.9) and PET is best with ERNIE (56.4). The abstract states the reverse pairing.
  • WSC: humans score 98%, while models stay near chance (~50%).
  • CSLDCP and stability: humans reach only 68% on CSLDCP, and PET comes within about 16 points of that. Results vary noticeably across splits.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Huilin Xu, Hu Yuan, Guoao Wei, Xiang Pan, Xin Tian, Libo Qin and Hu Hai. 2021. FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark. arXiv:2107.07498.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2107.07498 · arXiv 2107.07498

@misc{xu2021fewclue,
  title = {{FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark}},
  author = {Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Huilin Xu, Hu Yuan, Guoao Wei, Xiang Pan, Xin Tian, Libo Qin and Hu Hai},
  year = {2021},
  doi = {10.48550/arxiv.2107.07498},
  eprint = {2107.07498},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2107.07498}
}

References

28 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  2. Wei-Yu Chen, Yu-Chiang Frank Wang, Yen-Cheng Liu, Zsolt Kira, and Jia-Bin Huang. 2019. A CLOSER LOOK AT FEW-SHOT CLASSIFICA-TION. page 16. in Natural Language Processing. Association for Computational Linguistics. find
  3. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for Chinese natural language processing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 657–668, Online. Association for Computational Linguistics.
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  5. Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681–694. find
  6. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Association for Computational Linguistics (ACL).
  7. Hai Hu, Kyle Richardson, Xu Liang, Li Lu, Sandra Kübler, and Larry Moss. 2020. OCNLI: Original Chinese natural language inference. In Findings of Empirical Methods for Natural Language Processing (Findings of EMNLP).
  8. LTD. IFLYTEK CO. 2019. Iflytek: a multiple categories chinese text classifier. competition official website, http://challenge.xfyun.cn/2019/gamelist. ↗
  9. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  10. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. arXiv preprint arXiv:2103.10385.
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
  12. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Pre-training transformers as energy-based cloze models. In EMNLP. find
  13. Conversational-AI Center of OPPO XiaoBu. 2021. Bustm: Oppo xiaobu dialogue short text matching dataset. https://github.com/xiaobu-coai/BUSTM. ↗
  14. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sh… find
  15. Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Comp…
  16. Timo Schick and Hinrich Schütze. 2021b. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Assoc…
  17. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223.
  18. Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021. Improving and simplifying pattern exploiting training.
  19. Wilson L Taylor. 1953. “cloze procedure”: A new tool for measuring readability. Journalism quarterly, 30(4):415–433. find
  20. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Neural Information Processing Systems, pages 3266–3280.
  21. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
  22. Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner. find
  23. Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu. 2019. Nezha: Neural contextualized representation for chinese language understanding. find
  24. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
  25. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipe… — CLUE: A Chinese Language Understanding Evaluation Benchmark
  26. Zhe Zhao, Hui Chen, Jinbin Zhang, Wayne Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, and Xiaoyong Du. 2019. UER: An Open-Source Toolkit for Pre-training Models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confe…
  27. Chujie Zheng, Minlie Huang, and Aixin Sun. 2019. ChID: A Large-scale Chinese IDiom Dataset for Cloze Test. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 778–787.
  28. Yiming Yang Jaime Carbonell Russ R. Salakhutdinov Quoc V. Le Zhilin Yang, Zihang Dai. 2020. Xlnet: Generalized autoregressive pretraining for language understanding. In EMNLP.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2021-fewclue-chinese-few-shot/