📄 Arabic Humor as a Diagnostic Probe for Large Language Models

Summary

The paper argues that Arabic jokes make a sharp diagnostic for large language models. Understanding them calls on abilities that mainstream reasoning, knowledge and safety benchmarks rarely test: inferring speaker intent, reasoning about register in a diglossic language, and drawing on shared cultural background. Building on SSTH and GTVH, the author proposes a three-layer taxonomy of humor mechanisms (pragmatic, semantic, sociolinguistic). Thirteen curated social-media examples from five dialect regions illustrate it. Around the taxonomy sits an evaluation framework with three parts:

• contrastive minimal pairs,
• a multi-dimensional scoring rubric,
• an ontology of cultural presuppositions.

A small pilot probe of GPT-4o, Gemini 2.0 Flash and Claude Sonnet 4.5 finds four recurring failure types: sarcasm interpretation, reasoning about register contrast, coverage of dialect vocabulary, and cultural grounding. Jokes that switch from MSA to dialect produced the largest disagreement between models. The author presents the work as a proof of concept, not a benchmark. Its value is in showing how failures can be traced to specific missing capabilities, and in setting out a protocol for building larger annotated corpora.

Key points

• Taxonomy. It has three layers and nine mechanisms:
  • pragmatic: sarcasm/irony, self-deprecation, expectation inversion;
  • semantic: wordplay, metaphorical humor, hyperbole;
  • sociolinguistic: dialectal lexical humor, register switching, cultural references.
• Data. There are 13 examples from Twitter/X posts and memes (2023–2024) in Egyptian, Levantine, Gulf, Tunisian and Iraqi Arabic. Each was checked by the author and one colleague who knows the dialect. No inter-annotator agreement was measured.
• Models probed. GPT-4o, Gemini 2.0 Flash and Claude Sonnet 4.5. The register-switch examples (Examples 7 and 8) gave the strongest divergence between models.
• Context from related work.
  • ArSarcasm's BiLSTM baseline reached only F1 = 0.46.
  • In iSarcasmEval, MARBERT still struggled.
  • Over 90% of ChatGPT jokes came from just 25 templates (Jentzsch and Kersting, 2023).
  • ALPS documents "pragmatic literalism" in Arabic LLMs even in MSA.
• Stated limitations. Several varieties are missing (Moroccan Darija, Sudanese, Yemeni, Libyan). The study covers written text only. Transfer to other diglossic settings has not been tested.

Interesting ideas

• I1. Each joke gets one primary mechanism: the trigger whose removal kills the humor while the propositional content stays the same, and other triggers are logged as secondary amplifiers.
• I2. Contrastive minimal pairs keep the content fixed and change only one dimension, such as register, a dialect word or a cultural cue, so that a model's failure points to a single missing capability.
• I3. Language variety is treated as a carrier of meaning: a shift from formal setup to colloquial punchline is itself the signal, not just a label to detect.
• I4. A cultural presupposition ontology lists the background facts a joke silently assumes, such as a dish's social class associations.
• I5. Annotators are asked to surface disagreement rather than resolve it, treating variation across groups as a meaningful signal.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): Irony and implicature detection in prag could record a "necessary trigger" alongside each non-literal reading, so the explanation names the exact token or frame that flips the meaning.
• I2 (learn): Negative-control evaluation could auto-generate pairs that differ in one feature (e.g., a sarcastic vs. sincere continuation) and check that the pipeline's output changes only where it should.
• I3 (frames): A register or formality shift inside a text (formal narration followed by slangy reply) could be tagged as a pragmatic cue that feeds irony and deflation readings instead of being ignored after tagging.
• I4 (world): The compiled knowledge layer could store connotation and presupposition facts next to WordNet/Wikidata entries, letting the world model explain why a reference carries an implied contrast.
• I5 (learn): Where silver-data witness parsers or LLM teacher tiers disagree on pragmatic labels, the split could be kept as an "ambiguous" tag instead of being forced to one vote.