📄 A Comparative Evaluation of Semantic Ambiguity Detection in Two LLMs

Summary

This pilot study asks whether two OpenAI models, GPT-4.1 and its smaller sibling GPT-4.1 mini, notice when a text has more than one meaning, a skill tied to metalinguistic awareness. Drawing on test materials originally designed for children by Zipke et al. (2009) and by Shultz and Pilon (1973), the author built 29 contexts: riddles, single sentences, and sentence pairs. These cover lexical ambiguity and two kinds of structural ambiguity (surface-structure and deep-structure). Each context was paired with instructions at four levels, ranging from no hint of ambiguity to naming it outright, which gave 116 tasks per model. Responses were collected through the API with default settings and scored as pass/fail. Left unprompted, both models usually settled on a single reading. Their performance rose sharply once instructions suggested that multiple meanings might exist. The larger model did better on lexical items, while the mini model led on structural ones, so size alone did not predict success. The work is meant as groundwork for ambiguity-annotated datasets and for studying how LLMs handle newly coined, often ambiguous words.

Key points

• Task set: 8 riddles and 7 single sentences from Zipke et al. (2009), plus 14 sentence pairs from Shultz and Pilon (1973). These 29 contexts were run under 4 instruction levels, giving 116 tasks per model with binary scoring.
• Lexical ambiguity at Level 1 (no hint): GPT-4.1 solved 61% of tasks and GPT-4.1 mini only 11%. Neither model reached a perfect score even when ambiguity was stated directly.
• Deep-structure ambiguity: GPT-4.1 mini scored 50% at Level 1 and 100% on Levels 2–4. GPT-4.1 started at 17%, peaked at Level 3, and dropped at Level 4.
• Overall trend: GPT-4.1 mini improved with every level and overtook GPT-4.1 at Level 4. GPT-4.1 did not improve between Levels 3 and 4.
• Persistent failures and limitations: Both models failed every task built on two contexts. In SP4, "tank" was never read as a vehicle; in SP11, the vision-related sense of "see" was missed. The study also has clear limitations: lexical items are over-represented, there is no human baseline, and each task was run only once.