Unbounded Dependency Recovery for Parser Evaluation

What it is about

The authors build a parser-evaluation corpus of around 700 sentences annotated with unbounded dependencies in seven grammatical constructions. They run off-the-shelf parsers on it. The overall accuracy on these dependencies ranges from 25% to 59%. From this the authors argue that Parseval scores, which the authors find too forgiving about certain errors, are not a sufficient measure of parsing ability. They discuss why unbounded dependencies matter even though they are rare in corpora, and analyse the errors of the most successful parser, C&C. The corpus is the basis of the Nivre et al. (2010) study of dependency parsers.

Ideas

• I1. Mova construction corpus: about 700 sentences of our own across seven constructions (object and subject relatives, reduced relatives, wh-questions, tough adjectives, free relatives), held out from training; report recall per construction for en and gparse on each release candidate (dev).
• I2. en parse-eval as a construction gate: a release fails if any construction falls below its last-release value even when LAS rises; check that the gate goes red on a deliberately broken tagger (the negative control).
• I3. en/src/explain.rs: for each missed unbounded arc, print the transition where the parser lost the antecedent, and count the causes per construction on the dev set.

For Mova

Touches languages/en/25-relative-clauses.md, languages/en/26-questions.md and languages/en/23-complementation.md. This paper is the case for per-construction measures in Mova's gates. Build the corpus (I1) from our own annotation only, since the original corpus has a licence we cannot ship.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1699571.1699619

@inproceedings{rimell2009unbounded,
  title = {{Unbounded Dependency Recovery for Parser Evaluation}},
  author = {Rimell, Laura and Clark, Stephen and Steedman, Mark},
  booktitle = {Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  doi = {10.3115/1699571.1699619},
  url = {https://doi.org/10.3115/1699571.1699619}
}

References

26 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. E. Black, S. Abney, D. Flickenger, C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. A procedure for quantitatively comparing the syntactic coverage of English grammars. In HLT ’91: Proceedings of the Workshop on Speech and Natural Language, pages 306–311. find
2. Rens Bod. 2003. An efficient implementation of a new DOP model. In Proceedings of the 10th Meeting of the EACL, pages 19–26, Budapest, Hungary. find
3. Ted Briscoe, John Carroll, and Rebecca Watson. 2006. The second release of the RASP system. In Proceedings of the Interactive Demo Session of COLING/ACL-06, Sydney, Australia. find
4. A. Cahill, M. Burke, R. O’Donovan, J. van Genabith, and A. Way. 2004. Long-distance dependency resolution in automatically acquired wide-coverage PCFG-based LFG approximations. In Proceedings of the 42nd Meeting of the ACL, pages 320–327, Barcelona, Spain. find
5. Xavier Carreras, Michael Collins, and Terry Koo. 2008. Dynamic programming and the perceptron for efficient, feature-rich parsing. In Proceedings of the Twelfth Conference on Natural Language Learning (CoNLL-08), pages 9–16, Manchester, UK. find
6. John Carroll, Ted Briscoe, and Antonio Sanfilippo. 1998. Parser evaluation: a survey and a new proposal. In Proceedings of the 1st LREC Conference, pages 447–454, Granada, Spain. find
7. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Meeting of the ACL, pages 173–180, Michigan, Ann Arbor. find our card
8. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Meeting of the NAACL, pages 132–139, Seattle, WA. find
9. Stephen Clark and James R. Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. Computational Linguistics, 33(4):493–552. find our card
10. Stephen Clark, Mark Steedman, and James R. Curran. 2004. Object-extraction and question-parsing using CCG. In Proceedings of the EMNLP Conference, pages 111–118, Barcelona, Spain. find
11. Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of the 35th Meeting of the ACL, pages 16–23, Madrid, Spain. find
12. Michael Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. find our card
13. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th LREC Conference, Genoa, Italy. find our card
14. Daniel Gildea. 2001. Corpus variation and parser performance. In Proceedings of the 2001 EMNLP Conference, Pittsburgh, PA. find our card
15. Julia Hockenmaier and Mark Steedman. 2007. CCGbank: a corpus of CCG derivations and dependency structures extracted from the Penn Treebank. Computational Linguistics, 33(3):355–396. find our card
16. Liang Huang. 2008. Forest reranking: Discriminative parsing with non-local features. In Proceedings of the 46th Meeting of the ACL, pages 586–594, Columbus, Ohio. find
17. Mark Johnson. 2002. A simple pattern-matching algorithm for recovering empty nodes and their antecedents. In Proceedings of the 40th Meeting of the ACL, pages 136–143, Philadelphia, PA. find
18. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of Association for Computational Linguistics, pages 423–430, Sapporo, Japan. find our card
19. Roger Levy and Christopher Manning. 2004. Deep dependencies from context-free statistical parsers: correcting the surface dependency approximation. In Proceedings of the 42nd Meeting of the ACL, pages 328–335, Barcelona, Spain. find
20. Dekang Lin. 1995. A dependency-based method for evaluating broad-coverage parsers. In Proceedings of IJCAI-95, pages 1420–1425, Montreal, Canada. find
21. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the North American Chapter of the Association for Computational Linguistics Conference, pages 152–159, Brooklyn, NY. find our card
22. Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proceedings of the 43rd Meeting of the ACL, pages 91–98, Michigan, Ann Arbor. find our card
23. Yusuke Miyao and Jun’ichi Tsujii. 2005. Probabilistic disambiguation models for wide-coverage HPSG parsing. In Proceedings of the 43rd Meeting of the ACL, pages 83–90, Michigan, Ann Arbor. find our card
24. J. Nivre and M. Scholz. 2004. Deterministic dependency parsing of English text. In Proceedings of COLING-04, pages 64–70, Geneva, Switzerland. find our card
25. Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Proceedings of the HLT/NAACL Conference, Rochester, NY. find
26. Laura Rimell and Stephen Clark. 2008. Adapting a lexicalized-grammar parser to contrasting domains. In Proceedings of the 2008 EMNLP Conference, pages 475–484, Honolulu, Hawai‘i. find