📄 ARHAHA 2026: The Shared Task on Arabic Humor Automatic Generation

Summary

ARHAHA 2026 is a shared task on producing jokes in Arabic, an area that has received little research attention compared with humor detection or sarcasm classification. Each prompt gives systems two Arabic words, and the output must be a short joke of at most 100 characters that uses both of them. The joke must also be safe, benign, and original. The organizers hand-picked 1,200 word pairs across ten categories built to provoke incongruity. Evaluation runs in two steps. A Codabench script first checks format and word inclusion, and three annotators then judge 300 sampled outputs per system for humor, originality, lexical compliance, and safety. Nine teams registered, three submitted outputs, and two wrote system papers. All systems almost never produced unsafe or stereotyping text. Humor itself was rare: even the best system was judged funny in only about 5% of cases. The final scores were 49.25, 44.62, and 35.99. The results suggest that fluent text which satisfies the constraints is now easy to produce, while being genuinely funny is still the main bottleneck.

Key points

• The dataset has 1,200 manually curated Arabic word pairs, with 120 in each of ten semantic categories. Categories include logical contradiction, role reversal, scale/size violation, impossible behaviors, object misuse, workplace, technology, and the animal world.
• Each prompt carries three fields: pair ID, word1, and word2. Submissions are a CSV with id and generated_text, and output order must match prompt order exactly.
• The development phase (Dec 8, 2025 – Feb 9, 2026) checked format and constraints only. Humor counted only in the final phase (Feb 10–17, 2026), which was judged by humans.
• The top system was rated humorous in 5.01% of outputs. Final scores were 49.25, 44.62, and 35.99, and policy-violation and stereotyping rates stayed very low for all three systems.
• Two system papers were submitted:
  • Do3aba-AI used synthetic data distillation.
  • Arabic-HuCoT used ALLaM with HuCoT-based structured prompting.

Interesting ideas

• I1. Requiring a fixed pair of words in every output blocks systems from simply recalling memorized jokes, so the task measures real composition rather than retrieval.
• I2. Evaluation is split so that a cheap automatic gate (format, word presence) filters outputs before costly human judgment is spent on them.
• I3. Word inclusion is scored in three grades (both present, one, none) instead of pass/fail, which shows how far a system missed.
• I4. The error analysis finds "implicit substitution", where the model conveys a required concept through dialogue or a question without using the actual word.
• I5. Prompt categories are defined by the type of incongruity involved (role reversal, scale violation, impossible behavior, object misuse). Several failures came from choosing the wrong sense of a homograph, such as stadium read as professor.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): Generation tests could require that specific lemmas from the meaning graph surface in the output, proving that text is built from the graph rather than copied.
• I2 (learn): Mova's evaluation could apply the same gating, rejecting structurally broken outputs automatically before sending survivors to LLM-teacher or human review.
• I3 (answer): Answer checkers could grade how many required entities or values appear in a reply, giving partial-credit diagnostics instead of a binary score.
• I4 (back): Meaning-preserving but lexically unfaithful output could be flagged by comparing realized lemmas against the graph nodes that demanded explicit wording.
• I5 (frames): The irony/humor module could detect incongruity as a violation of expected roles or scales in the text's world model. It could also use the homograph failures as test cases for sense selection against frame context.