Elamur Research⌕
☰
home / papers / 2026 / XiaoM at SemEval-2026 Task 7: A Qwen-based System for Accurate Retrieval of Everyday Knowledge Across Diverse Languages and Cultures

XiaoM at SemEval-2026 Task 7: A Qwen-based System for Accurate Retrieval of Everyday Knowledge Across Diverse Languages and Cultures

Paper · 2026 · Xiao Yao and Liang Yang

Summary

This system paper addresses SemEval-2026 Task 7, which tests how well language models handle everyday cultural knowledge across 26 languages and 30 countries/regions using an extended BLEnD benchmark. The task has two tracks: short-answer questions (SAQ) and multiple-choice questions (MCQ). The authors use Qwen2.5-7B-Instruct without fine-tuning. Their work focuses on engineering a reliable inference pipeline rather than improving the model's cultural knowledge. The pipeline has five stages:

  • input normalization;
  • prompts that make cultural, geographic and linguistic cues explicit;
  • fully deterministic decoding;
  • retries on failed inference and on failed file writes;
  • rule-based calibration that forces outputs into the required TSV formats.

The paper's reported gains concern output consistency, failure rates and format compliance. It gives no official answer-accuracy figures. Its main value is as a practical template for producing valid, reproducible submissions under strict shared-task output constraints.

Key points

  • Model and benchmark: Qwen2.5-7B-Instruct with sampling disabled, temperature 0 and a repetition penalty. BLEnD data was excluded from training and served for evaluation only.
  • Consistency: Over 5 repeated runs on 100 samples per track, consistency rose from 32% (SAQ) and 64% (MCQ) under default sampling (temperature 0.7) to 100% on both tracks.
  • Retries: On 1000 samples, the inference failure rate fell from 0.8% with no retry to 0.5% with one retry and 0.1% with two.
  • Calibration: SAQ answers are truncated to 1–3 words, with a fallback to "no answer". MCQ outputs are restricted to A–D, with invalid outputs defaulting to A, and are then converted to binary vectors.
  • Format compliance: Compliance rose from 79.35% with no calibration to 82.45% with basic cleaning and 89.55% with full calibration. However, the text elsewhere claims 100% compliance on the final submission, an inconsistency the paper does not explain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dalian University of TechnologyDalian · China20 works · 46 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 552 works

Cite

Cite as Xiao Yao and Liang Yang. 2026. XiaoM at SemEval-2026 Task 7: A Qwen-based System for Accurate Retrieval of Everyday Knowledge Across Diverse Languages and Cultures. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.213

@inproceedings{yao2026xiaom,
  title = {{XiaoM at SemEval-2026 Task 7: A Qwen-based System for Accurate Retrieval of Everyday Knowledge Across Diverse Languages and Cultures}},
  author = {Xiao Yao and Liang Yang},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.213},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.213}
}

References

8 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Anacleto and 1 others. 2006. Can common sense uncover cultural differences? In Artificial Intelligence in Theory and Practice. find
  2. K. Arora and 1 others. 2022. Why exposure bias matters. In ACL Findings. find
  3. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
  4. D. Hershcovich and 1 others. 2022. Challenges in cross-cultural nlp. In ACL. find
  5. We performed inference under three settings: no calibration, basic cleaning, full calibration. Statistical results are shown in Table 3. find
  6. J. Myung, N. Lee, and Y. Zhou. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. In NeurIPS, volume 37. \#\#\# 5.5 Negative Results T.-P. Nguyen and 1 others. 2023. Extracting cultural commonsense knowledge at scale. In WWW.
  7. Ineffective strategies include enabling gradients during inference, weak generic prompts, removing fallback rules, and lacking deterministic decoding. Stability, prompt design, and standardization are all essential. F. Petroni and 1 others. 2019. Language models as knowledge bases? In EMNLP.
  8. D. Yin and 1 others. 2022. Geomlama: Geo-diverse commonsense probing. In EMNLP. find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yao-2026-xiaom-semeval-2026-task-7/