Elamur Research⌕
☰
home / papers / 2026 / CultRAG at SemEval-2026 Task 7: Hybrid Sparse-Dense Retrieval with Entity-Centric Knowledge Bases for Cultural MCQ Answering

CultRAG at SemEval-2026 Task 7: Hybrid Sparse-Dense Retrieval with Entity-Centric Knowledge Bases for Cultural MCQ Answering

Paper · 2026 · Aditya Singh and Rickarya Das

Summary

This paper presents CultRAG, a submission to SemEval-2026 Task 7 (BLEnD) Track 2. The track is a four-option multiple-choice benchmark of 47,014 English questions about everyday cultural knowledge across 30 country-language pairs. The authors asked whether retrieving from a curated cultural knowledge base could fill gaps in an LLM's knowledge of underrepresented cultures.

On top of Llama-3.1-8B-Instruct, they built a six-phase pipeline:

  • spaCy entity-based query expansion
  • hybrid BM25 + FAISS (all-MiniLM-L6-v2) retrieval combined via Reciprocal Rank Fusion
  • country filtering and keyword-based intent detection
  • tiered prompt routing and "anti-leak" prompting
  • reranking by source trust tier

The result is negative. The model without retrieval scored 78.6%, and the full system reached 78.5%, a gap McNemar's test finds insignificant (p = 0.962). An oracle check showed the KB contains the answer for only 40.7% of questions, well below the model's own accuracy. Only the anti-leak instructions recovered the losses caused by retrieval. Country-level results suggest retrieval helps poorly represented cultures but hurts well-covered ones, which argues for selective, coverage-aware RAG.

Key points

  • Knowledge base: 1,262 chunks in 7 shards drawn from Wikipedia, travel guides, news and country portals. Each source gets a trust weight of 1.0 (high), 0.6 (mid) or 0.3 (low) for reranking.
  • Ablation results:
    • Unfiltered RAG dropped accuracy to 77.9%.
    • Tiered routing (Phase 3) was the worst configuration at 77.6%.
    • Anti-leak filtering (Phase 4) brought it back to 78.5%.
    • Trust reranking and intent detection added nothing.
    • All effect sizes were negligible (Cohen's h ≤ 0.015).
  • KB coverage by country: it varies widely, from Bulgaria (69%) and Ethiopia (65%) down to Great Britain (21%).
  • RAG gains and losses:
    • Japan (+6.1pp) and Ethiopia (+4.1pp) gained most.
    • American Samoa (−6.2pp) and Great Britain (−3.0pp) lost most.
    • Overall, 14 countries improved and 14 got worse.
  • Official result: 77.55% accuracy on 29 of 30 locales, ranking 12th of 18 systems. Per-locale scores ranged from es-MX (94.00%) to am-ET (63.05%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology KharagpurKharagpur · India25 works · 714 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Information retrieval & search · 552 worksKnowledge graphs & ontologies · 211 works

Cite

Cite as Aditya Singh and Rickarya Das. 2026. CultRAG at SemEval-2026 Task 7: Hybrid Sparse-Dense Retrieval with Entity-Centric Knowledge Bases for Cultural MCQ Answering. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.393

@inproceedings{singh2026cultrag,
  title = {{CultRAG at SemEval-2026 Task 7: Hybrid Sparse-Dense Retrieval with Entity-Centric Knowledge Bases for Cultural MCQ Answering}},
  author = {Aditya Singh and Rickarya Das},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.393},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.393}
}

References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 758–759.
  2. Explosion. 2023. spACy: Industrial-strength natural language processing in Python. https://spacy.io. ↗
  3. Debanjan Ghosh, Kai North, Ekaterina Kochmar, Mamoru Komachi, and Marcos Zampieri, editors. 2026. Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). Association for Computational Linguistics, San Diego, United States. find
  4. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.
  5. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. find
  6. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. SimLM: Pre-training with representation bottleneck for document retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap… find
  7. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547. find
  8. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Pr…
  9. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
  10. Meta AI. 2024. Llama 3 model card. https://llama.meta.com/llama3. ↗
  11. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, and 1 others. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. Advances in Neural Information Processing Syste…
  12. Nedjma Ousidhoum, Junho Myung, Carla Perez-Almendros, Jiho Jin, Amr Keleg, Meriem Beloucif, Yi Zhou, Rodrigo Agerri, Vladimir Araujo, Naomi Baes, James Barry, Joanne Boisson, Nancy F. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2026-cultrag-semeval-2026-task-7/