Elamur Research⌕
☰
home / papers / 2026 / chengtang at SemEval-2026 Task 7: A Retrieval-Augmented Generation Framework for Cultural Perspective Alignment in Everyday MCQs

chengtang at SemEval-2026 Task 7: A Retrieval-Augmented Generation Framework for Cultural Perspective Alignment in Everyday MCQs

Paper · 2026 · Cheng Tang, Zhichao Meng and Meizhi Jin

Summary

This system paper addresses SemEval-2026 Task 7 (BLEnD), Track 2. Given an English multiple-choice question and a target language-region code, a system must choose which of four options fits that region's everyday culture. LLMs often default to English-centric assumptions on such questions.

The authors avoid fine-tuning and use a training-free RAG pipeline. They wrote short cultural profiles by hand for each locale, covering food, holidays, family, education, work and leisure. These profiles are indexed with text-embedding-v4, and the top three chunks are retrieved for each question. Generation is strictly zero-shot, with no BLEnD examples in the prompt. Two models, Gemini 3 Flash (preview) and GPT-5.2 Chat, answer each question in parallel, and a rule-based option-matching step picks the final answer.

The system scored 96.35 overall on roughly 47K evaluation items. Error analysis shows clear weak spots on Moroccan Arabic and on subjective Japanese everyday questions. The work suggests that lightweight retrieval of local knowledge helps cultural alignment, but it cannot fully make up for gaps in what the underlying model knows about dialects and local conventions.

Key points

  • Data:
    • Trial Data has 146 items across 23 locales.
    • Evaluation Data has 47,014 items with a strongly skewed language distribution, from Spanish at 4,807 items down to French at 307.
    • The evaluation set contains only 2,706 unique question stems, each reused on average 17.37 times and up to 1,144 times with different region codes.
  • Effect of RAG (Dev accuracy, with vs. without RAG):
    • Gemini 3 Flash: 97.97 vs. 94.59
    • GPT-5.2 Chat: 97.30 vs. 91.22
    • Claude Sonnet 4.5: 97.30 vs. 95.95
    • DeepSeek-V3.2: 88.51 vs. 84.46
  • Ensemble rules:
    • An output is accepted only if it matches one of the four options.
    • If both models give valid but different answers, Gemini's answer is used.
    • If neither output matches, the system picks the option whose embedding is most similar to the retrieved evidence.
    • Temperature is set to 0.0.
  • Weakest locales out of 30: ar-MA 83.98, ja-JP 87.32, en-AU 90.06 and ar-SA 91.89. Many locales, such as su-JB, bg-BG and as-AS, reach about 99%.
  • Diagnosed failure causes:
    • Moroccan Darija differs from other Arabic varieties because of Berber and French influence.
    • Some Japanese gold labels reflect local consensus rather than general real-world fact. In a convenience-store question, both models picked "train tickets" instead of the gold answer "rice ball."

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Cheng Tang, Zhichao Meng and Meizhi Jin. 2026. chengtang at SemEval-2026 Task 7: A Retrieval-Augmented Generation Framework for Cultural Perspective Alignment in Everyday MCQs. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.207

@inproceedings{tang2026chengtang,
  title = {{chengtang at SemEval-2026 Task 7: A Retrieval-Augmented Generation Framework for Cultural Perspective Alignment in Everyday MCQs}},
  author = {Cheng Tang, Zhichao Meng and Meizhi Jin},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.207},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.207}
}

References

9 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Claude sonnet 4.5 system card, September 2025. URL https://www.anthropic.com/system-cards. ↗
  2. Debanjan Ghosh, Kai North, Ekaterina Kochmar, Mamoru Komachi, and Marcos Zampieri, editors. Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, United States, July 2026. Association for Computational Linguistics. find
  3. Google DeepMind. Gemini 3 flash model card, December 2025. URL https://deepmind.google/ models/model-cards/gemini-3-flash/. ↗
  4. Md. Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, and Firoj Alam. NativQA: Multilingual culturally-aligned natural query for LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pile… find
  5. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseekv3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025.
  6. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. Advances in Neural Information Processing Systems, 37:78104–…
  7. OpenAI. Introducing GPT-5.2. Technical report, OpenAI, 2025. URL https://openai.com/index/ introducing-gpt-5-2/. ↗
  8. Nedjma Ousidhoum, Junho Myung, Carla Perez-Almendros, Jiho Jin, Amr Keleg, Meriem Beloucif, Yi Zhou, Rodrigo Agerri, Vladimir Araujo, Naomi Baes, James Barry, Joanne Boisson, Nancy F. Chen, Christine de Kock, Aleksandra Edwards, Joseba Fernandez de Landa, Mohamed Fazli Imam, Huda Hakami, Shu-Kai Hsi… — SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures
  9. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. Survey of cultural awareness in language models: Text and beyond. Computational Linguistics, 51(3):907–1004, 2025. doi: 10.1162/COLI.a. 14. URL ht…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2026-chengtang-semeval-2026-task-7/