Elamur Research⌕
☰
home / papers / 2026 / IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English

IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English

Paper · 2026 · Ana-Maria Luisa Mocanu, Sebastian Mocanu, Ciprian-Octavian Truica et al.

Summary

The paper addresses JOKER 2026 Task 1 (English): given a query describing a humor topic, a system must return the matching jokes, puns and wordplay from a corpus that mixes humorous and non-humorous short texts. The VANGUARD team built IROH, a three-stage pipeline. First, BM25 with query expansion and bge-base-en-v1.5 dense retrieval are merged by Reciprocal Rank Fusion. Second, a fine-tuned cross-encoder rescores the candidates. Third, an ensemble of LoRA-adapted LLM judges gives soft YES/NO scores. The judges are trained on one-sentence rationales written by Gemma 4 under two prompt styles: a generic one, and a typed one that names specific humor mechanisms. The system placed first on the leaderboard with 0.6347 MAP. The ablations are the more useful part. The distilled judge accounts for most of the ranking quality. Adding rationales to the index does almost nothing. Synthetic hard negatives raise local validation scores but lower official ones. A well-calibrated 7B judge outperforms every 31B setup. Together these findings suggest that training-data quality matters more than model size.

Key points

  • The pipeline narrows candidates in three steps:
    • RRF over BM25 and BGE keeps the top 4000.
    • GTE-Reranker-ModernBERT-Base cuts this to 1000.
    • A judge ensemble then scores those 1000: Qwen2.5-7B (QLoRA r=64) plus two Gemma-4-31B models (QLoRA r=32, one trained on generic and one on typed rationales), weighted 0.60, 0.30 and 0.10.
  • Scores from each stage are passed forward and mixed by linear interpolation of min-max normalised values. The optional confidence penalty at the end had no measurable effect.
  • The generic-rationale Qwen2.5-7B judge reached 0.6055 MAP on its own, beating every Gemma-4-31B configuration. Generic rationales outperformed typed ones, mainly for the smaller model.
  • Appending rationales to BM25 documents changed MAP from 0.2868 to 0.2876 (judge disabled), which is negligible. A grid of 195 ensemble weightings left the number of relevant retrieved documents unchanged, so recall is capped by the first stage.
  • The data analysis found that jokes are shorter than non-jokes (about 10.5 vs 18.9 words). They contain far more punctuation (r = 0.47 with the label) and use quotes much more often (56.2% vs 17.4%). These cues point to dialogue structure but do not separate the classes on their own.

Interesting ideas

  • I1. Rationales as supervision, not features. A teacher LLM produces a short explanation of whether, and why, each text counts as a joke matching the query. That text is used to train a smaller judge, not to enrich the index, and the paper shows only the training use pays off.
  • I2. Distrust gains that appear only on local validation. Synthetic hard negatives and rationale indexing both looked strong locally but did not hold up officially, which the authors attribute to distribution shift.
  • I3. Loose prompts can beat forced taxonomies. Making the teacher sort each joke into one of seven mechanism types made its rationales noisier, and the simpler generic prompt won.
  • I4. Dialogue structure as a humor cue. Quotes, punctuation density and patterns like "said Tom with fortitude" (Tom Swifties) correlate strongly with jokes.
  • I5. Diagnose the bottleneck by stage. Because relevant-retrieved counts stayed flat across all ensemble weightings, the authors could conclude that only first-stage recall limits further gains.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could attach a one-sentence justification to each silver label, so student components learn from explained decisions rather than bare labels.
  • I2 (learn): any augmentation or self-induced repair rule should be checked on a held-out, differently sourced set with negative controls before being kept.
  • I3 (frames): when collecting teacher data for irony and pun detection in prag, ask for free explanations first and assign mechanism categories afterwards, rather than forcing a type up front.
  • I4 (frames): a speech verb whose adverbial modifier relates semantically to the quoted clause is easy to detect on the UD tree and could flag pun candidates explainably.
  • I5 (answer): when QA readers fail, logging which stage lost the needed passage or entity shows whether to improve retrieval or reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitatea Națională de Știință și Tehnologie Politehnica BucureștiBucharest · Romania5 works · 265 citations🔬Academia Oamenilor de Știință din RomâniaBucharest · Romania3 works · 1 citation

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksPsychology · 1,649 worksSocial Psychology · 330 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEnglish · 2,379 worksInformation retrieval & search · 552 works

Cite

Cite as Ana-Maria Luisa Mocanu, Sebastian Mocanu, Ciprian-Octavian Truica and Elena-Simona Apostol. 2026. IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English. arXiv:2609.15618.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2609.15618 · arXiv 2609.15618

@misc{mocanu2026iroh,
  title = {{IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English}},
  author = {Ana-Maria Luisa Mocanu, Sebastian Mocanu, Ciprian-Octavian Truica and Elena-Simona Apostol},
  year = {2026},
  doi = {10.48550/arxiv.2609.15618},
  eprint = {2609.15618},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.15618}
}

References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. D. DiMartino, B. Konietzko, Avatar: The last airbender, 2006. Season 2, Episode 9: “Bitter Work”. find
  2. P. Vachharajani, et al., Overview of the CLEF JOKER Task 1: Humor-Aware Information Retrieval in English and Hinglish, in: E. S. Salido, A. Barrón-Cedeño, A. G. S. de Herrera, S. MacAvaney, J. M. Struß (Eds.), Working Notes of CLEF 2026: Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, CEUR-WS.org, 2026. find
  3. L. Ermakova, et al., Overview of the CLEF 2026 JOKER track: Humor detection, search, and translation, in: M. Hagen, M. Potthast, B. Stein, P. Schaer, E. Zangerle, S. MacAvaney, J. M. Struß, E. S. Salido, A. Barrón-Cedeño, A. G. S. de Herrera (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Seventeenth International Conference of the CLEF Association (CLEF 2026), Lecture Notes in Computer Science, Springer, 2026. find
  4. Google DeepMind, Gemma 4 model card, https://ai.google.dev/gemma/docs/core/model_card_4, 2026. Accessed: 2026-06-03. ↗
  5. Q. A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y.-C. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, Z. Qiu, S. Quan, Z. Wang, Qwen2.5 technical report, ArXiv abs/2412.15115 (2024). URL: https://api.semanticscholar.org/CorpusID:274859421. doi: https://doi.org/10.48550/arXiv.2412.15115 . ↗
  6. J. M. Suls, A two-stage model for the appreciation of jokes and cartoons: An information-processing analysis, The psychology of humor: Theoretical perspectives and empirical issues 1 (1972) 81–100. find
  7. O. Weller, K. Seppi, Humor detection: A transformer gets the last laugh, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 3621–3625. URL: https://aclanthology.org/D19-1372/. doi: 10.18653/v1/D19-1372 . ↗
  8. Y. Xie, J. Li, P. Pu, Uncertainty and surprisal jointly deliver the punchline: Exploiting incongruity-based features for humor recognition, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Association for Computational Linguistics, Online, 2021, pp. 33–39. URL: https://aclanthology.org/2021.acl-short.6/. doi: 10.18653/v1/2021.acl-short.6 . ↗
  9. L. Ermakova, A. Bosser, T. Miller, A. Jatowt, Overview of the CLEF 2024 JOKER task 1: Humour-aware information retrieval, in: G. Faggioli, N. Ferro, P. Galuscáková, A. G. S. de Herrera (Eds.), Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, 9-12 September, 2024, CEUR Workshop Proceedings, CEUR-WS.org, 2024, pp. 1775–1785. URL: https://ceur-ws.org/Vol-3740/paper-165.pdf. ↗
  10. L. Ermakova, R. Campos, A. Bosser, T. Miller, Overview of the CLEF 2025 JOKER lab: Humour in machine, in: J. Carrillo-de-Albornoz, A. G. S. de Herrera, J. Gonzalo, L. Plaza, J. Mothe, F. Piroi, P. Rosso, D. Spina, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction - 16th International Conference of the CLEF Association, CLEF 2025, Madrid, Spain, September 9-12, 2025, Proceedings, Lecture Notes in Computer Science, Springer, 2025, pp. 315–337. URL: https://doi.org/10.1007/978-3-032-04354-2_18. doi: 10.1007/978-3-032-04354-2\_18 . ↗
  11. E. Schuurman, M. Cazemier, L. Buijs, J. Kamps, University of amsterdam at the CLEF 2024 joker track, in: G. Faggioli, N. Ferro, P. Galuscáková, A. G. S. de Herrera (Eds.), Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, 9-12 September, 2024, CEUR Workshop Proceedings, CEUR-WS.org, 2024, pp. 1909–1922. URL: https://ceur-ws.org/Vol-3740/paper-181.pdf. ↗
  12. P. Vachharajani, pjmathematician at the CLEF 2025 JOKER lab tasks 1, 2 & 3: A unified approach to humour retrieval and translation using the qwen LLM family, in: G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), Working Notes of the Conference and Labs of the Evaluation Forum, CLEF 2025, Madrid, Spain, 9-12 September 2025, CEUR Workshop Proceedings, CEUR-WS.org, 2025, pp. 2889–2897. URL: https://ceur-ws.org/Vol-4038/paper_230.pdf. ↗
  13. R. Nogueira, K. Cho, Passage re-ranking with BERT, CoRR abs/1901.04085 (2019). URL: http://arxiv.org/abs/1901.04085. arXiv:1901.04085 . ↗
  14. H. Zhuang, Z. Qin, R. Jagerman, K. Hui, J. Ma, J. Lu, J. Ni, X. Wang, M. Bendersky, Rankt5: Fine-tuning T5 for text ranking with ranking losses, in: H. Chen, W. E. Duh, H. Huang, M. P. Kato, J. Mothe, B. Poblete (Eds.), Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, ACM, 2023, pp. 2308–2313. URL: https://doi.org/10.1145/3539618.3592047. doi: 10.1145/3539618.3592047 . ↗
  15. L. Xiong, C. Xiong, Y. Li, K. Tang, J. Liu, P. N. Bennett, J. Ahmed, A. Overwijk, Approximate nearest neighbor negative contrastive learning for dense text retrieval, in: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021. URL: https://openreview.net/forum?id=zeFrfgyZln. ↗
  16. X. Li, X. Li, H. Zhang, Z. Du, P. Jia, Y. Wang, X. Zhao, H. Guo, R. Tang, Syneg: Llm-driven synthetic hard-negatives for dense retrieval, CoRR abs/2412.17250 (2024). URL: https://doi.org/10.48550/arXiv.2412.17250. doi: 10.48550/ARXIV.2412.17250 . arXiv:2412.17250 . ↗
  17. H. A. Rahmani, E. Yilmaz, N. Craswell, B. Mitra, P. Thomas, C. L. A. Clarke, M. Aliannejadi, C. Siro, G. Faggioli, Llmjudge: Llms for relevance judgments, in: C. Siro, M. Aliannejadi, H. A. Rahmani, N. Craswell, C. L. A. Clarke, G. Faggioli, B. Mitra, P. Thomas, E. Yilmaz (Eds.), Proceedings of The First Workshop on Large Language Models for Evaluation in Information Retrieval (LLM4Eval 2024) co-located with 10th International Conference on Online Publishing (SIGIR 2024), Washington D.C., USA, July 18, 2024, CEUR Workshop Proceedings, CEUR-WS.org, 2024, pp. 1–3. URL: https://ceur-ws.org/Vol-3752/paper8.pdf. ↗
  18. T. Niu, S. Joty, Y. Liu, C. Xiong, Y. Zhou, S. Yavuz, Judgerank: Leveraging large language models for reasoning-intensive reranking, CoRR abs/2411.00142 (2024). URL: https://doi.org/10.48550/arXiv.2411.00142. doi: 10.48550/ARXIV.2411.00142 . arXiv:2411.00142 . ↗
  19. C.-Y. Hsieh, C.-L. Li, C.-k. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C.-Y. Lee, T. Pfister, Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Findings of the Association for Computational Linguistics: ACL 2023, Association for Computational Linguistics, Toronto, Canada, 2023, pp. 8003–8017. URL: https://aclanthology.org/2023.findings-acl.507/. doi: 10.18653/v1/2023.findings-acl.507 . ↗
  20. Y. Tian, Y. Han, X. Chen, W. Wang, N. V. Chawla, Beyond answers: Transferring reasoning capabilities to smaller llms using multi-teacher knowledge distillation, in: W. Nejdl, S. Auer, M. Cha, M. Moens, M. Najork (Eds.), Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM 2025, Hannover, Germany, March 10-14, 2025, ACM, 2025, pp. 251–260. URL: https://doi.org/10.1145/3701551.3703577. doi: 10.1145/3701551.3703577 . ↗
  21. A. Romanowski, P. H. V. Valois, K. Fukui, From punchlines to predictions: A metric to assess LLM performance in identifying humor in stand-up comedy, in: T. Kuribayashi, G. Rambelli, E. Takmaz, P. Wicke, J. Li, B.-D. Oh (Eds.), Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, Association for Computational Linguistics, Albuquerque, New Mexico, USA, 2025, pp. 36–46. URL: https://aclanthology.org/2025.cmcl-1.6/. doi: 10.18653/v1/2025.cmcl-1.6 . ↗
  22. H. Meghwani, A. Agarwal, P. Pattnayak, H. L. Patel, S. Panda, Hard negative mining for domain-specific retrieval in enterprise systems, in: G. Rehm, Y. Li (Eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), Association for Computational Linguistics, Vienna, Austria, 2025, pp. 1013–1026. URL: https://aclanthology.org/2025.acl-industry.72/. doi: 10.18653/v1/2025.acl-industry.72 . ↗
  23. A. Sinha, Don’t retrieve, generate: Prompting llms for synthetic training data in dense retrieval, CoRR abs/2504.21015 (2025). URL: https://doi.org/10.48550/arXiv.2504.21015. doi: 10.48550/ARXIV.2504.21015 . arXiv:2504.21015 . ↗
  24. K. Balog, D. Metzler, Z. Qin, Rankers, judges, and assistants: Towards understanding the interplay of llms in information retrieval evaluation, in: N. Ferro, M. Maistro, G. Pasi, O. Alonso, A. Trotman, S. Verberne (Eds.), Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2025, Padua, Italy, July 13-18, 2025, ACM, 2025, pp. 3865–3875. URL: https://doi.org/10.1145/3726302.3730348. doi: 10.1145/3726302.3730348 . ↗
  25. T. Miller, C. Hempelmann, I. Gurevych, SemEval-2017 task 7: Detection and interpretation of English puns, in: S. Bethard, M. Carpuat, M. Apidianaki, S. M. Mohammad, D. Cer, D. Jurgens (Eds.), Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Association for Computational Linguistics, Vancouver, Canada, 2017, pp. 58–68. URL: https://aclanthology.org/S17-2005/. doi: 10.18653/v1/S17-2005 . ↗
  26. G. Lessard, M. Levison, Computational model ling of linguistic humour: Tom swifty, in: Paper Delivered at the ALLC/ACH Joint Annual Conference. Christ Church, Oxford, 1992. find
  27. C. Kiddon, Y. Brun, That’s what she said: Double entendre identification, in: The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA - Short Papers, The Association for Computer Linguistics, 2011, pp. 89–94. URL: https://aclanthology.org/P11-2016/. ↗
  28. G. V. Cormack, C. L. A. Clarke, S. Büttcher, Reciprocal rank fusion outperforms condorcet and individual rank learning methods, in: J. Allan, J. A. Aslam, M. Sanderson, C. Zhai, J. Zobel (Eds.), Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19-23, 2009, ACM, 2009, pp. 758–759. URL: https://doi.org/10.1145/1571941.1572114. doi: 10.1145/1571941.1572114 . ↗
  29. S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, J. Nie, C-pack: Packed resources for general chinese embeddings, in: G. H. Yang, H. Wang, S. Han, C. Hauff, G. Zuccon, Y. Zhang (Eds.), Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, ACM, 2024, pp. 641–649. URL: https://doi.org/10.1145/3626772.3657878. doi: 10.1145/3626772.3657878 . ↗
  30. X. Zhang, Y. Zhang, D. Long, W. Xie, Z. Dai, J. Tang, H. Lin, B. Yang, P. Xie, F. Huang, M. Zhang, W. Li, M. Zhang, mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval, in: F. Dernoncourt, D. Preoţiuc-Pietro, A. Shimorina (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Association for Computational Linguistics, Miami, Florida, US, 2024, pp. 1393–1412. URL: https://aclanthology.org/2024.emnlp-industry.103/. doi: 10.18653/v1/2024.emnlp-industry.103 . ↗
  31. N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Association for Computational Linguistics, 2019, pp. 3980–3990. URL: https://doi.org/10.18653/v1/D19-1410. doi: 10.18653/V1/D19-1410 . ↗
  32. T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, Qlora: Efficient finetuning of quantized llms, in: A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, S. Levine (Eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL: http://papers.nips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html. ↗
  33. J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, Z. Liu, M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, in: L.-W. Ku, A. Martins, V. Srikumar (Eds.), Findings of the Association for Computational Linguistics: ACL 2024, Association for Computational Linguistics, Bangkok, Thailand, 2024, pp. 2318–2335. URL: https://aclanthology.org/2024.findings-acl.137/. doi: 10.18653/v1/2024.findings-acl.137 . ↗
  34. C. Li, M. Qin, S. Xiao, J. Chen, K. Luo, D. Lian, Y. Shao, Z. Liu, Making text embedders few-shot learners, in: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, 2025. URL: https://openreview.net/forum?id=wfLuiDjQ0u. A. Prompts ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mocanu-2026-iroh-insightful-ranking-humor-using/