Elamur Research⌕
☰
home / papers / 2016 / Evaluation Strategies for Computational Construction Grammars

Evaluation Strategies for Computational Construction Grammars

Summary

Implementations of computational construction grammars (ECG, FCG, TCG) have so far mostly been evaluated only through proof-of-concept demonstrations. As a result, there is effectively no way to compare grammar fragments across different platforms. The authors propose a set of metrics for reversible formalisms, which can both parse sentences into a formal meaning and produce sentences from a meaning. For parsing, they adapt Smatch from semantic parsing, since FCG meanings can be converted into AMR. To this they add a "specific" variant that evaluates the match only on the part of the meaning graph covered by the fragment under study. For production, they use a longest common substring (LCS) measure between the original and the reproduced sentence. They also propose profiling measures of search efficiency and grammar complexity. The metrics are tested on an FCG grammar that determines clitic placement in European Portuguese. The test set was compiled from examples in linguistic studies rather than from a newspaper corpus. It turned out that the general and specific S-scores can diverge noticeably. Such metrics help distinguish a broad-coverage grammar that fails on a particular phenomenon from a narrow precision grammar that handles that phenomenon well.

Key points

  • Two accuracy metrics are proposed: the S-score (Smatch) together with a specific variant for the meaning subgraph, and LCS for reproduced sentences. LCS is computed separately for meanings that passed all checks and for meanings obtained from a failed parse. Efficiency profiling is added separately.
  • The test set for Portuguese clitics contains 28 sentences with enclisis and 39 with proclisis. Proclisis in it is triggered by seven different factors: negation, wh-questions, relative clauses, fronted focus, operator adverbs, indefinite subjects, and downward-entailing quantifiers.
  • For enclisis, the general S-score is 0.75 and the specific one is 0.73. Hence, the clitic is misplaced more often than the general score would suggest.
  • The grammar performs worst at reproducing word order in relative clauses (LCS 0.43) and with operator adverbs (0.40). For the latter, Portuguese allows several correct orders, but only one was taken into account in the gold standard.
  • Efficiency scales poorly with sentence length: the branch leading to the solution accounts for less than 20% of the search nodes. A 3-word sentence requires on average about 9 constructions, and each additional word adds approximately 2 more.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Evaluation & benchmarks

Cite

Cite as Tânia Marques and Katrien Beuls. 2016. Evaluation Strategies for Computational Construction Grammars. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{marques2016evaluation,
  title = {{Evaluation Strategies for Computational Construction Grammars}},
  author = {Tânia Marques and Katrien Beuls},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  url = {https://aclanthology.org/C16-1108/}
}

References

28 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Victor Barrès and Jinyong Lee. 2014. Template construction grammar: from visual scene description to language comprehension and agrammatism. Neuroinformatics, 12(1):181–208. find
  2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Timothy Baldwin and Anna Korhonen, editors, Conference on Empirical Methods on Natural Language Processing, pages 1533–1544, Seattle, Washington. The Association for Computat… our card
  3. Benjamin Bergen and Nancy Chang. 2005. Embodied construction grammar in simulation-based language understanding. In Jan-Ola Östman and Mirjam Fried, editors, Construction grammars: Cognitive grounding and theoretical extensions, number 3 in Constructional Approaches to Language, pages 147–190. John… find
  4. Joan Bresnan, Ash Asudeh, Ida Toivonen, and Stephen Wechsler. 2016. Lexical-Functional syntax. Wiley-Blackwell, West-Sussex, England, 2 edition, August. find
  5. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, volume Volume 2: Short papers, pages 748–752, Sofia, Bulgaria, 4-9 August 2013. find
  6. Thomas H. Cormen. 2009. Introduction to algorithms. MIT Press, Boston, MA. find
  7. Charles J Fillmore, Paul Kay, and Mary Catherine O’Connor. 1988. Regularity and idiomaticity in grammatical constructions: The case of let alone. Language, 64(3):501–538. find
  8. Gerald Gazdar, Ewan Klein, Geoffrey K. Pullum, and Ivan A. Sag. 1985. Generalized Phrase Structure Grammar. Basil Blackwell, Oxford. find
  9. Adele Goldberg. 2005. Constructions at work. Oxford University Press, Oxford. find
  10. Thomas Hoffmann and Graeme Trousdale, editors. 2013. The Oxford handbook of Construction Grammar. Oxford University Press, Oxford. find
  11. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Main Volume, pages 605–612, Barcelona, Spain, July. A… our card
  12. Ana Luı́s and Ryo Otoguro. 2011. Inflectional morphology and syntax in correspondence: Evidence from european portuguese. In Glyn Hicks Alexandra Galani and George Tsoulas, editors, Morphology and Its Interfaces, number 178 in Linguistik Aktuell/Linguistics Today, pages 97–136. John Benjamins, Amste… find
  13. Ana Luı́s, Ryo Otoguro, Miriam Butt, and Tracy Holloway King. 2004. Proclitic contexts in european portuguese and their effect on clitic placement. In Miriam Butt and Tracy Holloway King, editors, The Proceedings of the LFG’04 Conference, pages 334–352, Stanford, CA. CSLI Publications. find
  14. Ana Maria Madeira. 1992. On clitic placement in European Portuguese. UCL Working Papers in Linguistics, 4:95–122. find
  15. Raymond J. Mooney. 2007. Learning for semantic parsing. In Alexander Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, volume 4394 of Lecture Notes in Computer Science, pages 311–324, Mexico City, Mexico, February 18-24. Springer. find
  16. Jan-Ola Östman and Mirjam Fried. 2005. Construction Grammars: Cognitive grounding and theoretical extensions, volume 3 of Constructional Approaches to Language. John Benjamins, Amsterdam. find
  17. Ankur P Parikh, Hoifung Poon, and Kristina Toutanova. 2015. Grounded semantic parsing for complex knowledge extraction. In Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 756–766, Denver, Colorado, June 5. Association for Computational Linguist… find
  18. Carl Pollard and Ivan A Sag. 1994. Head-driven phrase structure grammar. University of Chicago Press, Chicago. find
  19. Hoifung Poon and Pedro Domingos. 2009. Unsupervised semantic parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1, pages 1–10, Singapore, August 6-7. Association for Computational Linguistics. our card
  20. Nathan Schneider and Reut Tsarfaty. 2013. Book review: Design patterns in Fluid Construction Grammar. Computational Linguistics, 39(2):447–453. find
  21. Mark Steedman. 2000. The syntactic process. MIT Press, Boston, MA. find
  22. Luc Steels. 2004. Constructivist development of grounded construction grammars. In Walter Daelemans, editor, Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, pages 9–19, Barcelona. Association for Computational Linguistics. find
  23. Luc Steels, editor. 2011. Design patterns in Fluid Construction Grammar. Number 11 in Constructional Approaches to Language. John Benjamins, Amsterdam. find
  24. Luc Steels. to appear. The basics of Fluid Construction Grammar. Constructions and Frames. find
  25. Lappoon R Tang and Raymond J Mooney. 2001. Using multiple clause constructors in inductive logic programming for semantic parsing. In Proceedings of the 12th European Conference on Machine Learning, pages 466–477, Freiburg, Germany. Springer. find
  26. Sean Trott, Aurélien Appriou, Jerome Feldman, and Adam Janin. 2015. Natural language understanding and communication for multi-agent systems. In Artificial Intelligence for Human-Robot Interaction Papers from the AAAI 2015 Fall Symposium. AAAI. find
  27. Remi van Trijp. 2015. Cognitive vs. generative construction grammar: The case of coercion and argument structure. Cognitive Linguistics, 26(4):613–632. find
  28. Luke S. Zettlemoyer and Michael Collins. 2007. Online learning of relaxed ccg grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 678–687, Prague, June. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/marques-2016-evaluation-strategies-computational-construction-grammars/