Corpus Variation and Parser Performance

What it is about

The paper asks how much a statistical parser depends on the corpus it is trained on. It uses Collins' Model 1, a lexicalized generative parser, trained and tested on the Wall Street Journal (39,832 training sentences) and on the Brown corpus (about 21,800 training sentences, with every tenth sentence held out). A model trained on WSJ loses several points on Brown (recall 80.3 against 86.1 on WSJ), while a Brown-trained model does better on Brown than the WSJ model does. Removing the lexical-bigram parameters costs under 0.5 points on WSJ and leaves the cross-corpus results unchanged, so the bigrams are corpus-specific. Pruning the bigrams by a relative-entropy criterion removes 43% of the 736,000 parameters with a small loss.

Ideas

I2. Pruning curve as a size measure: prune lexical features by a count-times-divergence score to a fixed budget and report EWT test LAS against the number of kept features, for the shipped model size.

For Mova

I1. Lexical-feature ablation across banks: train en on EWT only, then on EWT plus ESLSpok, and test on ATIS (63.0 now) and GUM (76.85, test only) with and without lexical-identity features in en/src/parse.rs; the paper predicts that lexical pairs add little across corpora.
I3. Corpus-specific feature listing: after training, list the ten most weighted lexical feature pairs per bank, sorted deterministically with BTreeMap, to see which features are domain-specific before the model is shipped.

The code is en/src/parse.rs (features) and en/src/ptag.rs (its lexicon). The chapters are languages/en/19-lexicon-mwe-register.md and 37-register-and-variation.md. Measure first: the LAS drop on ATIS and GUM when lexical features are removed, since the paper shows that matched data beats more unmatched data.

Cite

Cite as Daniel Gildea. 2001. Corpus Variation and Parser Performance. Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{gildea2001corpus,
  title = {{Corpus Variation and Parser Performance}},
  author = {Daniel Gildea},
  booktitle = {Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing},
  year = {2001},
  url = {https://aclanthology.org/W01-0521/}
}

References

18 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Douglas Biber. 1993. Using register-diversi ed corpora for general language studies. Computational Linguistics , 19(2):219{241, June. find
Eugene Charniak. 1997. Statistical parsing with a context-free grammar and word statistics. In AAAI97 , Brown University, Providence, Rhode Island, August. find
Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Annual Meeting of the North American Chapter of the ACL (NAACL) , Seattle, Washington. our card
Michael Collins, Jan Hajic, Lance Ramshaw, and Christoph Tillmann. 1999. A statistical parser for czech. In Proceedings of the 37th Annual Meeting of the ACL , College Park, Maryland. our card
Michael Collins. 1996. A new statistical parser based on bigram lexical dependencies. In Proceedings of the 34th Annual Meeting of the ACL . find
Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of the 35th Annual Meeting of the ACL . our card
Michael Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing . Ph.D. thesis, University of Pennsylvania, Philadelphia. our card
Michael Collins. 2000. Discriminative reranking for natural language parsing. In Proceedings of the ICML . our card
Rebecca Hwa. 1999. Supervised grammar induction using training data with limited constituent information. In Proceedings of the 37th Annual Meeting of the ACL , College Park, Maryland. find
David Magerman. 1995. Statistical decision-tree models for parsing. In Proceedings of the 33rd Annual Meeting of the ACL . our card
Ivan A. Melcuk. 1988. Dependency Syntax: Theory and Practice . State University of New York Press. find
Adwait Ratnaparkhi. 1997. A linear observed time statistical parser based on maximum entropy models. In Proceedings of the Second Conference on Empirical Methods in Natural Language Processing . find
Douglas Roland and Daniel Jurafsky. 1998. How verb subcategorization frequencies are a ected by corpus choice. In Proceedings of COLING/ACL , pages 1122{1128. find
Douglas Roland, Daniel Jurafsky, Lise Menn, Susanne Gahl, Elizabeth Elder, and Chris Riddoch. 2000. Verb subcategorization frequency di erences between business-news and balanced corpora: the role of verb sense. In Proceedings of the Association for Computational Linguistics (ACL-2000) Workshop on Comparing Corpora . find
Kristie Seymore and Roni Rosenfeld. 1996. Scalable backo language models. In ICSLP-96 , volume 1, pages 232{235, Philadelphia. find
Daniel Sleator and Davy Temperley. 1993. Parsing english with a link grammar. In Third International Workshop on Parsing Technologies , August. our card
A. Stolcke, C. Chelba, D. Engle, V. Jimenez, L. Mangu, H. Printz, E. Ristad, R. Rosenfeld, D. Wu, F. Jelinek, and S. Khudanpur. 1996. Dependency language modeling. Summer Workshop Final Report 24, Center for Language and Speech Processing, Johns Hopkins University, Baltimore, April. find
Andreas Stolcke. 1998. Entropy-based pruning of backo language models. In Proc. DARPA Broadcast News Transcription and Understanding Workshop , pages 270{274, Lansdowne, Va. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gildea-2001-corpus-variation-parser-performance/

📄 Corpus Variation and Parser Performance