Elamur Research⌕
☰
home / papers / 2026 / Large Language Models Put to the Test on Chinese Noun Compounds: Experiments on Natural Language Inference and Compound Semantics

Large Language Models Put to the Test on Chinese Noun Compounds: Experiments on Natural Language Inference and Compound Semantics

Summary

Interpreting noun-noun compounds requires recovering an unstated link between modifier and head. LLMs have been tested on this mostly with English benchmarks. This paper evaluates how well current LLMs grasp Chinese compound semantics in two ways. The extrinsic test uses NCNLI, an inference dataset in which the correct label depends on understanding a compound in the premise. The intrinsic test uses NEPD, a new resource of noun compounds drawn from the New Era People's Daily Corpus. In NEPD, each compound carries a human-assigned relation from a fixed inventory of 11 categories plus OTHER.

The authors prompted four instruction-tuned ~7B Chinese models, Qwen-plus, DeepSeek-chat and GPT-4o-mini, in zero-shot and few-shot settings. They compared the results with fine-tuned BERT and RoBERTa baselines. On NLI, strong LLMs match or beat the supervised baselines, and the larger models benefit most from in-context examples. Choosing the exact relation is much harder: only Qwen-plus agrees with annotators in about a third of cases. The study offers a released Chinese benchmark and shows that LLM ability in this area is still limited.

Key points

  • NEPD dataset:
    • Candidate compounds were extracted with Jieba POS tagging and manually filtered, giving 2,083 items.
    • Three graduate students in linguistics labeled each compound's relation following the hierarchy of Liu and Liu (2019), whose eleven categories are, alphabetically, ABOUT, AND, BE, CAUSE, FOR, FROM, HAVE, IN, MAKE, OR and USE.
    • Items without a majority label were dropped, leaving 1,514 compounds.
    • Inter-annotator Jaccard overlap averaged 0.412, which is moderate agreement.
  • NCNLI: The dataset has 3,740 premise-hypothesis pairs, built from 66 templates applied to 625 compounds from Liu et al. (2022).
  • NLI results:
    • In zero-shot, Qwen2.5-7B scored highest (77.90% accuracy), above the fine-tuned RoBERTa baseline (72.94%).
    • With 3 shots, GPT-4o-mini reached the top score overall (80.41% accuracy).
    • The small models gained nothing consistent from the examples.
  • Relation interpretation:
    • Qwen-plus led with 36.13% accuracy, an R-rank of 4.42 and a Hit@5 of 73.34%.
    • GPT-4o-mini placed the correct relation in its top 5 often (69.22% Hit@5), but its exact-match accuracy was only 8.23%.
    • The 7B models mostly stayed near the random baseline of about 8.3%.
  • Relation difficulty:
    • In larger models, accuracy correlated weakly to moderately with relation frequency (Spearman up to 0.41 for Qwen-plus).
    • The rare relations OR and FROM were hardest (average accuracy under 3%).
    • CAUSE and MAKE were easiest (above 40%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mwe-1.1

@inproceedings{qiu2026large,
  title = {{Large Language Models Put to the Test on Chinese Noun Compounds: Experiments on Natural Language Inference and Compound Semantics}},
  author = {Qiu, Le and Chersoni, Emmanuele and Zhou, He and Hsu, Yu-Yin},
  booktitle = {Proceedings of the 22nd Workshop on Multiword Expressions (MWE 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mwe-1.1},
  url = {https://doi.org/10.18653/v1/2026.mwe-1.1}
}

References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Areej Alsini, Du Q Huynh, and Amitava Datta. 2020. Hit Ratio: An Evaluation Metric for Hashtag Recommendation. arXiv preprint arXiv:2010.01258. — [arXiv:2010.01258](https://arxiv.org/abs/2010.01258) find
  2. Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kuebler, and Larry Moss. 2020. OCNLI: Original Chinese Natural Language Inference. In Findings of EMNLP. find
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen Technical Report. arXiv preprint arXiv:2309.16609. find our card
  4. Shuiqing Huang and Dongbo Wang. 2019. Construction, Performance and Application of New Era People’s Daily Segmented Corpus (I) – Construction and Evaluation Corpus. Library and Information Service, 63(22):5–12. find
  5. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, and 1 others. 2024. Deepseek LLM: Scaling Open-source Language Models with Longtermism. arXiv preprint arXiv:2401.02954. find
  6. Gary Libben. 2014. The Nature of Compounds: A Psy-chocentric Perspective. Cognitive Neuropsychology, 31(1-2):8–25. find
  7. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In Proceedings of EMNLP. find
  8. Jingping Liu, Juntao Liu, Lihan Chen, Jiaqing Liang, Yanghua Xiao, Huimin Xu, Fubao Zhang, Zongyu Wang, and Rui Xie. 2022. Noun Compound Interpretation with Relation Classification and Paraphrasing. IEEE Transactions on Knowledge and Data Engineering, 35(9):8757–8769. find
  9. Cristina Butnariu, Su Nam Kim, Preslav Nakov, Diarmuid O Séaghdha, Stan Szpakowicz, and Tony Veale. 2009. SemEval-2010 Task 9: The Interpretation of Noun Compounds Using Paraphrasing Verbs and Prepositions. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Direct… find
  10. Pengyuan Liu and Yujie Liu. 2019. Semantic Relations Hierarchy and Knowledge Base Construction of Chinese Basic Noun Compounds. Journal of Chinese Information Processing, 33(4):20–28. find
  11. Jose Camacho-Collados, Claudio Delli Bovi, Luis Espinosa Anke, Sergio Oramas, Tommaso Pasini, Enrico Santus, Vered Shwartz, Roberto Navigli, and Horacio Saggion. 2018. SemEval-2018 Task 9: Hypernym Discovery. In Proceedings of SemEval. find
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. find
  13. Please identify the semantic relation between the premise (P) and the hypothesis (H) and respond with one of the following semantic relations: Entailment, Contradiction or Neutral. Return their relation only. P: xxx H: xxx Preslav Nakov. 2008a. Noun Compound Interpretation Using Paraphrasing Verbs:… find
  14. Preslav Nakov. 2008b. Paraphrasing Verbs for Noun Compound Interpretation. In Proceedings of the LREC Workshop on Multiword Expressions. find
  15. In the few-shot setting, we concatenate a representative example for each category to the template above, while ensuring that none of them overlaps with the evaluation set. find
  16. Giulia Rambelli, Emmanuele Chersoni, Claudia Collacciani, and Marianna Bolognesi. 2024. Can Large Language Models Interpret Noun-Noun Compounds? A Linguistically-Motivated Study on Lexicalized and Novel Compounds. In Proceedings of ACL. find
  17. 前提: 运动员有一个不锈钢饭盒。 假设: 不锈钢是饭盒的制作材料。 输出: 蕴含 Vered Shwartz. 2019. A Systematic Comparison of English Noun Compound Representations. In Proceedings of the ACL Workshop on Multiword Expressions. find
  18. P: The athlete has a stainless steel lunch box. H: Stainless steel is the material that the lunch box is made of. Output: Entailment Vered Shwartz and Chris Waterson. 2018. Olive Oil Is Made of Olives, Baby Oil Is Made for Babies: Interpreting Noun Compounds Using Paraphrases in a Neural Model. In P… find
  19. Henk J Van Jaarsveld and Gilbert E Rattink. 1988. Frequency Effects in the Processing of Lexicalized and Novel Nominal Compounds. Journal of Psycholinguistic Research, 17:447–473. find
  20. 前提: 清洁工昨天吃了巧克力蛋糕。 假设: 清洁工吃的蛋糕里没有巧克 力。 输出: 矛盾 Meng Wang, CR Huang, Shiwen Yu, Bin Li, and 1 others. 2010. Chinese Noun Compound Interpretation based on Paraphrasing Verbs. (Journal of Chinese Information Processing), 24(6):3–9. find
  21. P: The janitor ate chocolate cake yesterday. H: There was no chocolate in the cake that the janitor ate. Output: Contradiction Meng Wang, Lulu Wang, Na Tian, and Bin Li. 2016. Automatic Interpretation of Chinese Noun Compounds Based on Word Similarity. ICIC Express Letters, Part B: Applications, 7(6… find
  22. 前提: 科学家最喜欢的是椒盐牛蛙。 假设: 科学家只吃过椒盐口味的牛 蛙。 输出: 中立 An Yang, Baosong Yang, B Zhang, B Hui, B Zheng, B Yu, Chengpeng Li, D Liu, F Huang, H Wei, and 1 others. 2024. Qwen2.5 Technical Teport. arXiv preprint arXiv:2412.15115. find
  23. P: The scientist’s favorite is salt-and-pepper bullfrogs. H: The scientist has only eaten bullfrogs with a salt-and-pepper flavor. Output: Neutral He Zhou, Yu Yin Hsu, and Emmanuele Chersoni. 2024. Evaluating Chinese Noun Compound Interpretation in Natural Language Inference. In Proceedings of the C… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qiu-2026-large-language-models-put-test/