基于预训练模型的文本常识合理性估计研究综述

Summary

This survey covers commonsense plausibility estimation, also called commonsense validation. The task asks whether a complete statement agrees with everyday human experience and shared knowledge. The authors distinguish it from fact verification, which relies on explicit evidence and gives definite answers, and from commonsense QA, which fills in missing information. Plausibility estimation instead verifies a whole statement. The authors argue it matters for catching commonsense errors in LLM output, which word-overlap metrics fail to detect.

They screened about 150 papers published between 2018 and 2024 and organised pretrained-model methods along two axes:

• Knowledge source: the model's internal knowledge versus external augmentation.
• Fine-tuning dependency: supervised training versus zero-/few-shot use.

The survey also catalogues 48 evaluation datasets, grouped by task format and by type of commonsense knowledge. It closes with three open directions: zero-shot complex reasoning, fusion of heterogeneous knowledge, and multimodal commonsense. Overall, it is a structured map of the field that covers the post-2023 LLM era, which earlier surveys did not reach.

Key points

• Literature search: sources were ACL Anthology, IEEE Xplore, ACM Digital Library, Google Scholar, arXiv and OpenReview, with emphasis on venues such as ACL, EMNLP, NAACL, AAAI, NeurIPS and ICLR.
• Three task formats:
  • Binary labels, evaluated with accuracy, F1 and AUC-ROC.
  • Continuous scores, evaluated with Pearson's γ and Spearman's ρ.
  • Distributions over candidate concepts, evaluated with Spearman, KL divergence and Top-N accuracy.
• Internal-knowledge methods:
  • Classifiers built on encoder representations.
  • Likelihood scoring adapted to each pre-training objective: autoregressive (GPT), masked-token (BERT) and replaced-token detection (ELECTRA).
  • Prompting instruction-following LLMs, optionally with chain-of-thought or generated background knowledge.
• Known weakness of likelihood scoring: it suffers from surface-form competition. Proposed fixes include adding domain information and reweighting keywords (e.g., CASE).
• Historical lineage: rule-based systems came first (CYC, started in 1984), followed by knowledge bases (ConceptNet, WordNet) and statistical learning, before the shift to pretrained models.