📄 If Probable, Then Acceptable? Understanding Conditional Acceptability Judgments in Large Language Models

Summary

The paper asks whether large language models judge "If A, then B" statements the way people do. In humans, such judgments depend on two things: how likely B is once A is assumed, and whether A actually bears on B (supports it, undermines it, or has nothing to do with it). The authors reuse the human-rated materials of Skovgaard-Olsen et al. (2016), which consist of 144 conditionals set in everyday scenarios. They give these items to Llama 3.1 and Qwen 2.5 models of two sizes each, under vanilla, few-shot and CoT prompting. Each model provides three kinds of ratings: conditional probability, probability of the whole conditional, and acceptability. Linear mixed-effects models and ANOVA tests then measure how strongly each factor drives the ratings.

Nearly all models respond to both probability and semantic relevance. However, they combine the two less systematically than humans, and their behavior shifts considerably with prompt style and model size. Bigger models do not track human patterns more closely. These results matter for anyone relying on LLMs to evaluate hypothetical or causal claims.

Key points

• Data: 144 conditionals spread over 12 contexts. Each context crosses 3 relation types (POS, NEG, IRR) with 4 prior-probability combinations for antecedent and consequent. Human ratings come from crowdsourcing, and coverage is uneven, ranging from roughly 3 to 10 raters per item.
• Models: Llama 3.1 (8B, 70B) and Qwen 2.5 (7B, 72B). CoT was applied only to the larger models. Every task asked for a single score from 0 to 100.
• Exception: Llama 70B with vanilla prompting was the one setup that did not incorporate both conditional probability and relevance.
• Prompting effects: Few-shot examples made models more sensitive to relevance but could bias them toward particular relation types. Humans consistently rated positively related conditionals highest, and models only partly reproduced this.
• Alternative elicitation: Correlations with directly elicited ratings were weakly positive for sentence probability (about 0.07–0.33) and weakly to moderately negative for perplexity (about −0.08 to −0.38). These metrics therefore capture something different from direct elicitation.