Elamur Research⌕
☰
home / papers / 2026 / G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment

G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment

Paper · 2026 · Fengying Ye, Yanming Sun, Runzhe Zhan et al.

Summary

Idioms carry figurative, culture-bound meanings that cannot be built word by word, so models that map them literally across languages often fail. Existing idiom datasets mostly cover detection or literal-vs-figurative disambiguation. They do not offer a unified, diagnostic test of idiom-to-idiom equivalence. The authors built G-IdiomAlign to fill this gap.

Each idiom in the benchmark is tied to an English Wiktionary gloss, which acts as a shared meaning pivot. Reference pairs come from a precision-oriented pipeline:

  • keep only single-sense entries,
  • retrieve candidates by gloss embeddings,
  • enforce bidirectional one-to-one matches,
  • apply pair-specific, distribution-aware thresholds.

Two protocols are defined. The first is multiple-choice equivalence with typed distractors, so errors can be attributed. The second is open generation that compares prompts with and without the gloss.

Across many LLMs, the dominant error is choosing literal renderings, and it is worst for low-resource targets. Supplying glosses reliably helps generation, but scores stay modest. An attention study on Qwen3-8B links successful gloss-aided outputs to stronger focus on the gloss. The benchmark therefore exposes real gaps in models' cultural-semantic competence.

Key points

  • Coverage: 18,785 idiom pairs spanning 36 language pairs drawn from a main set of nine languages: German, English, Spanish, Finnish, French, Japanese, Polish, Chinese and Portuguese. Arabic, Korean, Thai and Vietnamese are reported separately in an appendix. En–Zh is the largest pair (1,782 pairs) and Fr–Ja the smallest (186).
  • Construction: glosses are embedded with OpenAI text-embedding-3-large. The pipeline retrieves the top-10 candidates per idiom, then retains only mutual nearest neighbor (MNN) rank-1 pairs. Polysemous idioms are removed.
  • Tasks: Multiple-Choice Idiom Equivalence uses LLM-generated typed distractors. Gloss-Contrastive Generation (No-gloss vs. With-gloss) is scored with a fixed-threshold embedding-similarity proxy.
  • Qwen3-8B attention diagnostics (n = 189 annotated outputs):
    • Correct With-gloss generations show a higher gloss attention ratio (GAR) and lower DR and OtherTop1 (q < 0.05).
    • Idiom-span attention (IAR) does not separate correct from wrong outputs (q = 0.250).
    • Differences between conditions sit mainly in attention heads rather than whole layers.
  • Limitations: the English-pivot design may introduce English-centric bias. The one-to-one constraint leaves out synonym clusters. The attention findings are correlational and come from a single model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MacauMacao · China33 works · 2,223 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 worksIdioms & multiword expressions · 217 works

Cite

Cite as Fengying Ye, Yanming Sun, Runzhe Zhan, Lidia S. Chao, Zheqi Zhang and Derek F. Wong. 2026. G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1794

@inproceedings{ye2026g,
  title = {{G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment}},
  author = {Fengying Ye, Yanming Sun, Runzhe Zhan, Lidia S. Chao, Zheqi Zhang and Derek F. Wong},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1794},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1794}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude Haiku 4.5. find
  2. Francesca De Luca Fornaciari, Begoña Altuna, Itziar Gonzalez-Dios, and Maite Melero. 2024. A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models. In Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024), pages 35–44, Mexico City, Mexico (Hybrid). A…
  3. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. DeepSeek-V3.2: Pushing the Frontier of Open…
  4. Qiuyu Ding, Hailong Cao, and Tiejun Zhao. 2024. Enhancing bilingual lexicon induction via bi-directional translation pair retrieving. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17898– 17906. find
  5. Sundesh Donthi, Maximilian Spencer, Om B. Patel, Joon Young Doh, Eid Rodan, Kevin Zhu, and Sean O’Brien. 2025. Improving LLM abilities in idiomatic translation. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 175–181, Abu Dhabi, United Arab Emirates. Associa… find
  6. Maggie Mi, Aline Villavicencio, and Nafise Sadat Moosavi. 2025. Rolling the DICE on idiomaticity: How LLMs fail to grasp context. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7314–7332, Vienna, Austria. Association for Comp…
  7. Samin Fakharian and Paul Cook. 2021. Contextualized embeddings encode monolingual and cross-lingual knowledge of idiomaticity. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 23–32, Online. Association for Computational Linguistics. find
  8. Mistral AI team. 2024. Un ministral, des ministraux. find
  9. Yicheng Fu, Zhemin Huang, Liuxin Yang, Yumeng Lu, and Zhongdongming Dai. 2025. CHENGYU-BENCH: Benchmarking large language models for Chinese idiom understanding and use. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2355–2366, Suzhou, China. Associa… find
  10. Diego Moussallem, Mohamed Ahmed Sherif, Diego Esteves, Marcos Zampieri, and Axel-Cyrille Ngonga Ngomo. 2018. LIdioms: A multilingual linked idioms data set. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Re… find
  11. Google Gemini Team. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.
  12. OpenAI. 2024. New embedding models and API updates. find
  13. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 279–287, Marseille, France. European Language Resources Association.
  14. Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2025. Investigating idiomaticity in word representations. Computational Linguistics, 51:505–555.
  15. Paria Khoshtab, Danial Namazifard, Mostafa Masoudi, Ali Akhgary, Samin Mahdizadeh Sani, and Yadollah Yaghoobzadeh. 2025. Comparative study of multilingual idioms and similes in large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8680–8698, A…
  16. Shuang Li, Jiangjie Chen, Siyu Yuan, Xinyi Wu, Hao Yang, Shimin Tao, and Yanghua Xiao. 2024. Translate meanings, not just words: IdiomKB’s role in optimizing idiomatic translation with language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):18554–18563. find
  17. Yaoyiran Li, Anna Korhonen, and Ivan Vulić. 2023. On bilingual lexicon induction with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9577–9599, Singapore. Association for Computational Linguistics. find
  18. Emmy Liu, Aditi Chaudhary, and Graham Neubig. 2023. Crossing the threshold: Idiomatic machine translation through retrieval augmentation and loss weighting. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15095–15111, Singapore. Association for Comput…
  19. OpenAI. 2026. GPT-5.1: A smarter, more conversational ChatGPT. find
  20. Ming Qian. 2024. Automating idiom translation with cross-lingual natural language generation grounded in semantic analyses using large language models. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations), pages 95–115, Chicago, U… find
  21. Prateek Saxena and Soma Paul. 2020. EPIE Dataset: A Corpus For Possible Idiomatic Expressions. Preprint, arXiv:2006.09479. — [arXiv:2006.09479](https://arxiv.org/abs/2006.09479) find
  22. Uliana Sentsova, Debora Ciminari, Josef Van Genabith, and Cristina España-Bonet. 2025. MultiCoPIE: A multilingual corpus of potentially idiomatic expressions for cross-lingual PIE disambiguation. In Proceedings of the 21st Workshop on Multiword Expressions (MWE 2025), pages 67–81, Albuquerque, New M…
  23. Yanming Sun, Runzhe Zhan, Chi Seng Cheang, Han Wu, Xuebo Liu, Yuyao Niu, Fengying Ye, Kaixin Lan, Lidia S Chao, and Derek F Wong. 2026. Exposing the cracks: Vulnerabilities of retrieval-augmented llm-based machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume… find
  24. Qwen Team. 2025. Qwen3-Max: Just Scale it. find
  25. Simone Tedeschi, Federico Martelli, and Roberto Navigli. 2022. ID10M: Idiom identification in 10 languages. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2715–2726, Seattle, United States. Association for Computational Linguistics.
  26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  27. An Yang, Junyang Lin, Jingren Zhou, and et al. 2025b. Qwen3-embedding-8B. find
  28. Cai Yang, Yao Dou, David Heineman, Xiaofeng Wu, and Wei Xu. 2025c. Evaluating LLMs on chinese idiom translation. ArXiv, abs/2508.10421. find
  29. Jianing Zhou, Hongyu Gong, and Suma Bhat. 2021. PIE: A parallel idiomatic expression corpus for idiomatic sentence generation and paraphrasing. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 33–48, Online. Association for Computational Linguistics. find
  30. Jianing Zhou, Ziheng Zeng, and Suma Bhat. 2023. CLCL: Non-compositional expression detection with contrastive learning and curriculum learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 730–743, Toronto, Canada. Associat… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ye-2026-g-idiomalign-gloss-pivoted-benchmark/