Elamur Research⌕
☰
home / papers / 2007 / Validation and Evaluation of Automatically Acquired Multiword Expressions for Grammar Engineering

Validation and Evaluation of Automatically Acquired Multiword Expressions for Grammar Engineering

Paper · 2007 · Aline Villavicencio, Valia Kordoni, Yi Zhang et al.

Summary

The paper addresses how to validate and evaluate multiword expressions (MWEs) acquired automatically to extend a precise HPSG grammar of English. The authors took 1039 trigram candidates found by the error mining system of Zhang et al. (2006). On these, they compared three statistical measures: MI, χ² and permutation entropy (PE). They tested whether these measures can distinguish genuine MWEs from random word sequences. Frequencies were counted in four sources: a fragment of the BNC, the full BNC, and via Google and Yahoo. The authors then abandoned the words-with-spaces approach in favour of a compositional one. They heuristically identify the head word of the expression, and a lexical type predictor trained on the Redwoods treebank generates new entries for it in the ERG lexicon. It turned out that MI and PE separate MWEs from non-MWEs. Web data rank frequencies almost the same way as the BNC: their large size compensates for their lack of balance. The compositional extension yielded the same coverage gain as the previous work, but with far fewer new entries and without loss of parsing accuracy. This matters for preserving the quality of hand-built grammars.

Key points

  • On a BNC subcorpus of 674 sentences, ERG coverage rose from 7.1% to 22.7%: the PET parser parsed 153 sentences instead of 48.
  • For 30 candidates, 21 new lexical entries sufficed (≈0.7 per expression). In Zhang et al. (2006), 311 candidates required 373 entries (≈1.2 per expression), and coverage there rose from 5% to 18%.
  • The proportion of parsed sentences with at least one correct analysis rose from 66.7% to 81.0%. This is close to the ~83% typical of the ERG on the BNC. The disambiguation model places the best analysis in the top-5 in 66% of cases and in the top-10 in 75%.
  • Agreement in trigram rankings is highest between Google and Yahoo (Q = 0.86). It exceeds the agreement between the BNC fragment and the full BNC. The Kendall's τ correlation between all corpora is significant (p<0.000001).
  • Motivation for the work: according to Baldwin et al. (2004), ~40% of ERG failures on the BNC are caused by missing lexical entries, and missing MWEs account for 8% of all errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations🎓Universidade Federal do Rio Grande do SulPorto Alegre · Brazil8 works · 522 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksSyntax & grammar formalisms · 214 worksIdioms & multiword expressions · 217 works

Cite

Cite as Aline Villavicencio, Valia Kordoni, Yi Zhang, Marco Idiart and Carlos Ramisch. 2007. Validation and Evaluation of Automatically Acquired Multiword Expressions for Grammar Engineering. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{villavicencio2007validation,
  title = {{Validation and Evaluation of Automatically Acquired Multiword Expressions for Grammar Engineering}},
  author = {Aline Villavicencio, Valia Kordoni, Yi Zhang, Marco Idiart and Carlos Ramisch},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1110/}
}

References

21 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Timothy Baldwin and Aline Villavicencio. 2002. Extracting the unextractable: A case study on verb-particles. In Proc. of the 6th Conference on Natural Language Learning (CoNLL-2002), Taipei, Taiwan. find
  2. Timothy Baldwin, Emily M. Bender, Dan Flickinger, Ara Kim, and Stephan Oepen. 2004. Road-testing the English Resource Grammar over the British National Corpus. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004), Lisbon, Portugal.
  3. Gosse Bouma, Gertjan van Noord, and Robert Malouf. 2001. Alpino: Wide-coverage computational analysis of dutch. In Computational Linguistics in The Netherlands 2000. find
  4. Ted Briscoe and John Carroll. 1997. Automatic extraction of subcategorization from corpora. In Fifth Conference on Applied Natural Language Processing, Washington, USA.
  5. Ulrich Callmeier. 2001. Efficient parsing with large-scale unification grammars. Master’s thesis, Universität des Saarlandes, Saarbrücken, Germany. find
  6. Stefan Evert and Brigitte Krenn. 2005. Using small random samples for the manual evaluation of statistical association measures. Computer Speech and Language, 19(4):450–466. find
  7. Dan Flickinger. 2000. On building a more efficient grammar by exploiting types. Natural Language Engineering, 6(1):15–28.
  8. Ray Jackendoff. 1997. Twistin’ the night away. Language, 73:534–59. find
  9. Jeremy Nicholson and Timothy Baldwin. 2006. Interpretation of compound nominalisations using avg. analysis \# 335.08 285.01 coverage % 7.1% 22.7% corpus and web statistics. In Proceedings of the Workshop on Multiword Expressions: Identifying and Exploiting Underlying Properties, pages 54– 61, Sydney,… find
  10. Stephan Oepen, Kristina Toutanova, Stuart Shieber, Christopher Manning, Dan Flickinger, and Thorsten Brants. 2002. The LinGO Redwoods treebank: Motivation and preliminary applications. In Proceedings of COLING 2002: The 17th International Conference on Computational Linguistics: Project Notes, Taipe…
  11. Stephan Oepen. 2001. [incr tsdb()] — competence and performance laboratory. User manual. Technical report, Computational Linguistics, Saarland University, Saarbrücken, Germany. find
  12. Darren Pearce. 2002. A comparative evaluation of collocation extraction techniques. In Third International Conference on Language Resources and Evaluation, Las Palmas, Canary Islands, Spain. find
  13. William H. Press, Saul A. Teukolsky, William T. Vet-terling, and Brian P. Flannery. 1992. Numerical Recipes in C: The Art of Scientific Computing. Second edition. Cambridge University Press. find
  14. Ivan Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In Proceedings of the 3rd International Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2002), pages 1–15, Mexico City, Mexico.
  15. Kristina Toutanova, Christoper D. Manning, Stuart M. Shieber, Dan Flickinger, and Stephan Oepen. 2002. Parse ranking for a rich HPSG grammar. In Proceedings of the First Workshop on Treebanks and Linguistic Theories (TLT2002), pages 253–263, Sozopol, Bulgaria. find
  16. Leonoor van der Beek. 2005. The extraction of determinerless pps. In Proceedings of the ACL-SIGSEM Workshop on The Linguistic Dimensions of Prepositions and their Use in Computational Linguistics Formalisms and Applications, Colchester, UK. find
  17. Gertjan van Noord. 2004. Error mining for wide-coverage grammar engineering. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Main Volume, pages 446–453, Barcelona, Spain, July. find
  18. Aline Villavicencio, Francis Bond, Anna Korhonen, and Diana McCarthy. 2005. Introduction to the special issue on multiword expressions: having a crack at a hard nut. Journal of Computer Speech and Language Processing, 19(4):365–377. find
  19. Aline Villavicencio. 2005. The availability of verb-particle constructions in lexical resources: How much is enough? Journal of Computer Speech and Language Processing, 19. find
  20. Yi Zhang and Valia Kordoni. 2006. Automated deep lexical acquisition for robust open texts processing. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006), Genoa, Italy. find
  21. Yi Zhang, Valia Kordoni, Aline Villavicencio, and Marco Idiart. 2006. Automated multiword expression prediction for grammar engineering. In Proceedings of the Workshop on Multiword Expressions: Identifying and Exploiting Underlying Properties, pages 36–44, Sydney, Australia. Association for Computat…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/villavicencio-2007-validation-evaluation-automatically-acquired-multiword/