Elamur Research⌕
☰
home / papers / 2026 / Semantic Parsing for Evaluating Large Language Models: Separating Linguistic Abilities with YARN

Semantic Parsing for Evaluating Large Language Models: Separating Linguistic Abilities with YARN

Summary

Off-the-shelf LLMs struggle to produce formal meaning representations. Earlier evaluations usually use one notation, often AMR, and a single aggregate graph-matching score. This makes it unclear whether errors come from the format or from particular semantic phenomena, and well-known formalisms risk data contamination. The authors address this by having LLMs parse sentences into Yarn, a layered formalism derived from AMR. Yarn separates predicate–argument structure from feature layers such as tense, aspect, modality, negation and quantification. Outputs are scored with SmatchY, an extension of Smatch that can restrict scoring to chosen element types or layers. The experiments vary three factors:

  • inference mode (standard vs. reasoning)
  • linearization (a JSON graph encoding vs. a logic-style CFG notation)
  • whether partial semantic parses are given as auxiliary hints

No configuration wins consistently, and results depend heavily on format and supervision. Extra semantic hints help some models but hurt others. Tense and negation are recovered more reliably than quantification. The approach shows which aspects of meaning LLMs capture and can guide the design of LLM-based semantic parsing pipelines.

Key points

  • Reference data: 100 Yarn annotations of short sentences from the Parallel Universal Dependencies (PUD) English treebank (UD 1.0). The sentences average 7.22 words and range from 3 to 10 words. Five additional human-annotated sentences serve as few-shot examples.
  • The annotations are released at github.com/YARN-World and can be browsed on semantics.grew.fr with GREW queries.
  • SmatchY aligns atomic elements of the gold and candidate graphs, then computes precision, recall and F1. Scoring can be filtered by predicates, arguments, feature modifiers or specific layers. One example is SmatchY-Fol, which covers predicates, arguments, negation and quantification.
  • SmatchY is purely structural. Logically equivalent but structurally different annotations, such as reordered commuting quantifiers, count as mismatches, so the score is a conservative lower bound.
  • The two linearizations contrast familiarity with novelty. JSON is common in pretraining data, while the CFG notation is more compact, shows operator scope explicitly and is unfamiliar to LLMs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4hqtuv5e47pw

@inproceedings{vergnette2026semantic,
  title = {{Semantic Parsing for Evaluating Large Language Models: Separating Linguistic Abilities with YARN}},
  author = {VERGNETTE, Rémi DE and Amblard, Maxime},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4hqtuv5e47pw},
  url = {https://doi.org/10.63317/4hqtuv5e47pw}
}

References

29 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discou… find our card
  2. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2023. Prompting language models for linguistic structure. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6649– 6663, Toronto, Canada. Association for Computational Linguistics. find
  3. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. find
  4. Aaron Conklin, Michael Frank, and Ellie Pavlick. 2023. What do llms know about linguistic theory? arXiv preprint arXiv:2310.01672. — [arXiv:2310.01672](https://arxiv.org/abs/2310.01672) find
  5. Rémi de Vergnette, Maxime Amblard, and Bruno Guillaume. 2025. Evaluation framework for layered meaning representation. In Proceedings of the Sixth International Workshop on Designing Meaning Representations, pages 38–48, Prague, Czechia. Association for Computational Lingustics. find
  6. Yanai Elazar, Atticus Geiger, Ellie Pavlick, and Yoav Goldberg. 2021. Measuring and improving compositional generalization in language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL). find
  7. Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics. Transactions of the Association for Computational Linguistics, 8:34–48. find
  8. Allyson Ettinger, Jena Hwang, Valentina Pyatkin, Chandra Bhagavatula, and Yejin Choi. 2023. “you are an expert linguistic annotator”: Limits of LLMs as analyzers of Abstract Meaning Representation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8250–8263, Singapore.… find
  9. Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A Discriminative Graph-Based Parser for the Abstract Meaning Representation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426–1436, Balti… find
  10. Gottlob Frege. 1967. Begriffsschrift, a formula language, modeled upon that of arithmetic, for pure thought. In Jean van Heijenoort, editor, From Frege to Gödel: A Source Book in Mathematical Logic, 1879–1931, pages 1–82. Harvard University Press. Originally published in 1879. find
  11. Bruno Guillaume, Guillame Bonfante, Paul Masson, Mathieu Morey, and Guy Perrier. 2012. Grew : un outil de réécriture de graphes pour le TAL (Grew: a graph rewriting tool for NLP) [in French]. In Proceedings of the Joint Conference JEP-TALN-RECITAL 2012, volume 5: Software Demonstrations, pages 1–2,… find
  12. Aishwarya Kamath and Rajarshi Das. 2018. A survey on semantic parsing. arXiv preprint arXiv:1812.00978. — [arXiv:1812.00978](https://arxiv.org/abs/1812.00978) find
  13. Paul Kingsbury and Martha Palmer. 2002. From TreeBank to PropBank. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02), Las Palmas, Canary Islands - Spain. European Language Resources Association (ELRA). find our card
  14. Percy Liang. 2016. Learning executable semantic parsers for natural language understanding. Commun. ACM, 59(9):68–76. find
  15. Meta AI. 2025. The llama 4 herd: The beginning of a new era of natively multimodal intelligence. Meta AI Blog. find
  16. Mistral AI. 2023. Mixtral of experts. Blog post. find
  17. Siyana Pavlova. 2025. Tools and methods for semantically annotated corpora. Ph.D. thesis, Université de Lorraine. find
  18. Siyana Pavlova, Maxime Amblard, and Bruno Guillaume. 2024. YARN is All You Knit: Encoding Multiple Semantic Phenomena with Layers. In Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024, pages 66–76, Torino, Italia. ELRA and ICCL. find
  19. Phillip Schneider, Manuel Klettner, Kristiina Jokinen, Elena Simperl, and Florian Matthes. 2024. Evaluating large language models in semantic parsing for conversational question answering over knowledge graphs. find
  20. Wesley Scivetti, Melissa Torgbi, Austin Blodgett, Mollie Shichman, Taylor Hudson, Claire Bonial, and Harish Tayyar Madabushi. 2025. Beyond memorization: Assessing semantic generalization in large language models using phrasal constructions. find
  21. Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2025. A survey on employing large language models for text-to-sql tasks. ACM Comput. Surv., 58(2). find
  22. John F. Sowa. 1984. Conceptual Structures: Information Processing in Mind and Machine. Addison-Wesley, Reading, MA. find
  23. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392. find
  24. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  25. Shira Wein and Juri Opitz. 2024. A survey of AMR applications. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6856–6875, Miami, Florida, USA. Association for Computational Linguistics. find
  26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… find our card
  27. Yu Yuan, Lili Zhao, Kai Zhang, Guangting Zheng, and Qi Liu. 2024. Do LLMs overcome shortcut learning? an evaluation of shortcut challenges in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12188–12200, Miami, Florida, USA. Asso… find
  28. Zeyu Zhan, E. Haihong, and Min Song. 2025. Leveraging large language models for enhanced text-to-sql parsing. IEEE Access. 7. find
  29. Language Resource References Nivre, Joakim and Bosco, Cristina and Choi, Jinho and de Marneffe, Marie-Catherine and Dozat, Timothy and Farkas, Richárd and Foster, Jennifer and Ginter, Filip and Goldberg, Yoav and Hajič, Jan and Kanerva, Jenna and Laippala, Veronika and Lenci, Alessandro and Lynn, Te… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vergnette-2026-semantic-parsing-evaluating-large-language/