Elamur Research⌕
☰
home / papers / 2026 / Pinetree at SemEval-2026 Task 7: A Large-Scale Failure Analysis of Cultural Grounding in Language Models

Pinetree at SemEval-2026 Task 7: A Large-Scale Failure Analysis of Cultural Grounding in Language Models

Paper · 2026 · Yen Yee Yam and Hong Meng Yam

Summary

The paper asks how well large language models know routine, majority-default daily practices across cultures. This area has received less study than alignment measured with value surveys. For SemEval-2026 Task 7 Track 2 (47,014 multiple-choice questions across 30 locales, extending BLEnD), the authors queried a frozen GPT-5.2 with one structured prompt at temperature 0. They used no fine-tuning, retrieval, chain-of-thought or ensembling. The prompt steered the model toward contemporary, nationally mainstream behaviour and away from stereotypes and surface word overlap. This reached 88.85% micro and 90.55% macro accuracy, placing fourth.

The main contribution is an analysis of all 5,241 wrong answers. Each was labelled with its BLEnD topic via question-ID lookup and tagged with rule-based cross-cutting flags. Relative to topic size, errors peak in Family and Work life. Western brand defaults are rare and confined to essentially one sports-team template. Numeric mistakes usually land one option away from the gold answer. The authors conclude that everyday cultural misalignment stems from sparse local knowledge rather than ideological bias.

Key points

  • Results: 88.85% micro and 90.55% macro accuracy (rank #4). Country accuracy spans 69.96% (am-ET) to 99.28% (es-EC), and the weakest locales include ha-NG, ko-KP, fa-IR and ar-MA.
  • By absolute share, errors fall mostly in Food (39.42%), but within-topic error rates are highest for Family (21.04%) and Work life (20.45%). Rates are lowest for Sport (4.87%) and Holidays/Celebration/Leisure (6.90%).
  • Brand-attractor errors are only 2.50% of failures (131 cases):
    • 129 come from a "most popular sport team" question in four low-resource cultures.
    • 2 come from Australian payment-method questions.
  • Of 1,180 numeric errors, 71.1% (n=839) are rank-adjacent to the correct option, suggesting approximate rather than missing knowledge.
  • Heuristic error flags matched an independent GPT-5.2 blind labeler on 90.91% of the 4,061 non-numeric errors. The labeler confirmed 117 of the 131 brand cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,053 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Yen Yee Yam and Hong Meng Yam. 2026. Pinetree at SemEval-2026 Task 7: A Large-Scale Failure Analysis of Cultural Grounding in Language Models. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.422

@inproceedings{yam2026pinetree,
  title = {{Pinetree at SemEval-2026 Task 7: A Large-Scale Failure Analysis of Cultural Grounding in Language Models}},
  author = {Yen Yee Yam and Hong Meng Yam},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.422},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.422}
}

References

7 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  2. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. Preprint, arXiv:2303.17466.
  3. Yoshinari Fujinuma, Siddharth Varia, Nishant Sankaran, Srikar Appalaraju, Bonan Min, and Yogarshi Vyas. 2023. A multi-modal multilingual benchmark for document image classification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14361–14376. find
  4. Debanjan Ghosh, Kai North, Ekaterina Kochmar, Mamoru Komachi, and Marcos Zampieri, editors. 2026. Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). Association for Computational Linguistics, San Diego, United States. find
  5. Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  6. Candace Ross, Boris Katz, and Andrei Barbu. 2021. Measuring social biases in grounded vision and language embeddings. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 998–1008, Online. Association… find
  7. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9):pgae346.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yam-2026-pinetree-semeval-2026-task-7/