Elamur Research
☰
home / papers / 2001 / Corpus Variation and Parser Performance

Corpus Variation and Parser Performance

What it is about

The paper asks how much a statistical parser depends on the corpus it is trained on. It uses Collins' Model 1, a lexicalized generative parser, trained and tested on the Wall Street Journal (39,832 training sentences) and on the Brown corpus (about 21,800 training sentences, with every tenth sentence held out). A model trained on WSJ loses several points on Brown (recall 80.3 against 86.1 on WSJ), while a Brown-trained model does better on Brown than the WSJ model does. Removing the lexical-bigram parameters costs under 0.5 points on WSJ and leaves the cross-corpus results unchanged, so the bigrams are corpus-specific. Pruning the bigrams by a relative-entropy criterion removes 43% of the 736,000 parameters with a small loss.

Ideas

For Mova

The code is en/src/parse.rs (features) and en/src/ptag.rs (its lexicon). The chapters are languages/en/19-lexicon-mwe-register.md and 37-register-and-variation.md. Measure first: the LAS drop on ATIS and GUM when lexical features are removed, since the paper shows that matched data beats more unmatched data.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W01-0521.pdf

Summary

Most statistical parsing research had been trained and evaluated only on the Wall Street Journal (WSJ) section of the Penn Treebank. Gildea asks how well such parsers transfer to other text, and which parts of the model depend most on the training corpus. He reimplements Collins (1997) Model 1, a lexicalized generative history-based parser, and runs cross-corpus experiments on WSJ and the Treebank II portion of the Brown corpus, which is mostly fiction. He trains on WSJ, on Brown, and on both combined.

Three findings stand out:

  • Training data: Matched in-domain training data matters far more than sheer quantity. A WSJ-trained model degrades sharply on Brown.
  • Lexical bigrams: Head-word pair statistics are highly corpus-specific. Dropping them barely changes in-domain accuracy and does not change cross-domain accuracy at all.
  • Pruning: Adapting Seymore–Rosenfeld weighted-difference pruning from speech language modeling shrinks the parser model by up to 43% with only a small loss.

The work matters for domain adaptation, for interpreting WSJ benchmark results, and for building memory-efficient parsers.

Key points

  • Data:

    • WSJ uses the standard split: sections 2–21 for training (39,832 sentences) and section 23 for testing.
    • The Brown training set has 21,818 sentences. Every tenth sentence was held out for testing.
    • Both test sets only include sentences of 40 words or fewer.

  • Cross-corpus results (labeled constituent recall/precision):

    TrainingTestRecall/Precision
    WSJWSJ86.1/86.6
    WSJBrown80.3/81.0
    BrownBrown83.6/84.6
    WSJ+BrownBrown83.9/84.8

    Adding mismatched data helped by less than 0.5% absolute.

  • Removing lexical bigrams:

    • WSJ in-domain performance fell only to 85.6/86.2.
    • WSJ→Brown performance stayed exactly at 80.3/81.0.

  • Model size and pruning:

    • The full model has about 736K parameters across nine distributions.
    • Pruning removed between 96K parameters (13%) and 316K (43%), with recall dropping only from 86.1 to 85.6.
    • Pruned models fit within 256MB of memory.

  • Top-scoring bigrams:

    • In WSJ they are finance-specific base NP pairs such as "New York" and "Wall Street".
    • In Brown they are generic verb–subject and preposition–object pairs.

Interesting ideas

  • Ablate one component under a cross-domain split. Removing the most specific feature and checking in-domain versus out-of-domain scores shows whether that feature captures general structure or memorizes the corpus. This diagnostic transfers to any feature-rich model.
  • Borrow pruning from language modeling. Any backoff-structured model can use the weighted-difference criterion: count multiplied by the log-probability change after backing off. Parameters are dropped when rarely used or already well approximated by the coarser estimate.
  • Tune size with a single threshold. One threshold gives a smooth size/accuracy trade-off instead of an all-or-nothing choice. This is useful for deploying models on limited hardware.
  • Compare top-scoring parameters across corpora. Normalizing the pruning score by corpus size makes the highest-scoring parameters comparable between corpora. Inspecting them reveals domain-specific vocabulary, a cheap way to characterize genre differences.
  • Use interleaved test sampling for heterogeneous corpora. Holding out every n-th sentence makes the test set more representative of a mixed corpus. Its drawback is document overlap between training and test, which likely makes the task look easier and should be reported.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{gildea2001corpus,
  title = {{Corpus Variation and Parser Performance}},
  author = {Gildea, Daniel},
  booktitle = {Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing},
  year = {2001},
  url = {https://aclanthology.org/W01-0521/}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Douglas Biber. 1993. Using register-diversi ed corpora for general language studies. Computational Linguistics , 19(2):219{241, June. find
  2. Eugene Charniak. 1997. Statistical parsing with a context-free grammar and word statistics. In AAAI97 , Brown University, Providence, Rhode Island, August. find
  3. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Annual Meeting of the North American Chapter of the ACL (NAACL) , Seattle, Washington. find
  4. Michael Collins, Jan Hajic, Lance Ramshaw, and Christoph Tillmann. 1999. A statistical parser for czech. In Proceedings of the 37th Annual Meeting of the ACL , College Park, Maryland. find
  5. Michael Collins. 1996. A new statistical parser based on bigram lexical dependencies. In Proceedings of the 34th Annual Meeting of the ACL . find
  6. Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of the 35th Annual Meeting of the ACL . find
  7. Michael Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing . Ph.D. thesis, University of Pennsylvania, Philadelphia. find our card
  8. Michael Collins. 2000. Discriminative reranking for natural language parsing. In Proceedings of the ICML . find our card
  9. Rebecca Hwa. 1999. Supervised grammar induction using training data with limited constituent information. In Proceedings of the 37th Annual Meeting of the ACL , College Park, Maryland. find
  10. David Magerman. 1995. Statistical decision-tree models for parsing. In Proceedings of the 33rd Annual Meeting of the ACL . find
  11. Ivan A. Melcuk. 1988. Dependency Syntax: Theory and Practice . State University of New York Press. find
  12. Adwait Ratnaparkhi. 1997. A linear observed time statistical parser based on maximum entropy models. In Proceedings of the Second Conference on Empirical Methods in Natural Language Processing . find
  13. Douglas Roland and Daniel Jurafsky. 1998. How verb subcategorization frequencies are a ected by corpus choice. In Proceedings of COLING/ACL , pages 1122{1128. find
  14. Douglas Roland, Daniel Jurafsky, Lise Menn, Susanne Gahl, Elizabeth Elder, and Chris Riddoch. 2000. Verb subcategorization frequency di erences between business-news and balanced corpora: the role of verb sense. In Proceedings of the Association for Computational Linguistics (ACL-2000) Workshop on Comparing Corpora . find
  15. Kristie Seymore and Roni Rosenfeld. 1996. Scalable backo language models. In ICSLP-96 , volume 1, pages 232{235, Philadelphia. find
  16. Daniel Sleator and Davy Temperley. 1993. Parsing english with a link grammar. In Third International Workshop on Parsing Technologies , August. find our card
  17. A. Stolcke, C. Chelba, D. Engle, V. Jimenez, L. Mangu, H. Printz, E. Ristad, R. Rosenfeld, D. Wu, F. Jelinek, and S. Khudanpur. 1996. Dependency language modeling. Summer Workshop Final Report 24, Center for Language and Speech Processing, Johns Hopkins University, Baltimore, April. find
  18. Andreas Stolcke. 1998. Entropy-based pruning of backo language models. In Proc. DARPA Broadcast News Transcription and Understanding Workshop , pages 270{274, Lansdowne, Va. find