Elamur Research⌕
☰
home / papers / 2004 / Intricacies of Collins' Parsing Model

Intricacies of Collins' Parsing Model

Summary

Collins' lexicalized generative parsing models set accuracy benchmarks on the Penn Treebank and became the basis for much later work. Researchers who rebuilt them from the published descriptions, however, could not match the reported scores. Bikel built a multilingual engine for generative statistical parsing, set it up as Collins' Model 2, and found that a version built only from the thesis fell well short on Section 00 of the Penn Treebank. Through reverse engineering, study of Collins' code and discussion with Collins, he documented every unpublished detail needed to reproduce the benchmarks. These include an extensive training-tree preprocessing pipeline and smoothing and back-off choices. Leaving them out raises relative error by 11%.

He also proposes a simpler way to handle punctuation and conjunction that performs equally well. He then re-examines familiar features. Bilexical dependencies contribute very little, and head choice matters less than people assumed. Most of the benefit of lexicalization seems to come from generating unlexicalized structure conditioned on the headword and its POS tag. The work matters for replicability and for guiding the design of newer parsing models.

Key points

  • A clean-room implementation of Collins' Model 2 based only on published details has 11% more relative error than the full-detail version. An earlier Model 1 reimplementation by Gildea (2001) showed about 16.7% more relative error.
  • The paper lists 11 ordered preprocessing steps for training trees. Among them are:
    • base-NP (NPB) insertion and repair
    • null-element removal
    • punctuation raising
    • argument identification
    • head-finding
  • It refines the definitions of coordinated phrases and of when to insert an extra NP level, beyond what Collins' 1999 thesis specifies.
  • Bilexical dependencies are rarely used by the model, and head choice has limited impact on parsing accuracy.
  • The parsing engine developed for this work (Bikel 2002) is publicly available, and the evaluation uses Penn Treebank Section 00 as the development set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Treebanks

Cited by

104 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 92 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/0891201042544929

@article{bikel2004intricacies,
  title = {{Intricacies of Collins' Parsing Model}},
  author = {Bikel, Daniel M.},
  journal = {Computational Linguistics},
  year = {2004},
  doi = {10.1162/0891201042544929},
  url = {https://doi.org/10.1162/0891201042544929}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baker, J. K. 1979. Trainable grammars for speech recognition. In Spring Conference of the Acoustical Society of America, pages 547–550, Boston. find
  2. Bies, A. 1995. Bracketing guidelines for Treebank II style Penn Treebank Project. Available at ftp://ftp.cis.upenn.edu/pub/ treebank/doc/manual/root.ps.gz. find
  3. Bikel, Daniel M. 2000. A statistical model for parsing and word-sense disambiguation. In Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, Hong Kong, October. find
  4. Bikel, Daniel M. 2002. Design of a multi-lingual, parallel-processing statistical parsing engine. In Proceedings of HLT2002, San Diego. find
  5. Bikel, Daniel M. and David Chiang. 2000. Two statistical parsing models applied to the Chinese Treebank. In Martha Palmer, Mitch Marcus, Aravind Joshi, and Fei Xia, editors, Proceedings of the Second Chinese Language Processing Workshop, pages 1–6, Hong Kong. find
  6. Bikel, Daniel M., Richard Schwartz, Ralph Weischedel, and Scott Miller. 1997. Nymble: A high-performance learning name-finder. In Fifth Conference on Applied Natural Language Processing, pages 194–201, Washington, DC. find our card
  7. Black, Ezra, Frederick Jelinek, John Lafferty, David Magerman, Robert Mercer, and Salim Roukos. 1992. Towards history-based grammars: Using richer models for probabilistic parsing. In Proceedings of the Fifth DARPA Speech and Natural Language Workshop, Harriman, NY. find
  8. Booth, T. L. and R. A. Thompson. 1973. Applying probability measures to abstract languages. IEEE Transactions on Computers, volume C-22: 442–450. find
  9. Chiang, David and Daniel M. Bikel. 2002. Recovering latent information in treebanks. In Proceedings of COLING’02, Taipei. find
  10. Cohen, Paul R. 1995. Empirical Methods for Artificial Intelligence. MIT Press, Cambridge, MA. find
  11. Collins, Michael. 1996. A new statistical parser based on bigram lexical dependencies. In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pages 184–191, Santa Cruz, CA. find
  12. Collins, Michael. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of ACL-EACL ’97, pages 16–23, Madrid. find our card
  13. Collins, Michael. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. find
  14. Collins, Michael. 2000. Discriminative reranking for natural language parsing. In International Conference on Machine Learning, Stanford University, Stanford, CA. find our card
  15. Collins, Michael and Nigel Duffy. 2002. New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In Proceedings of ACL-02, pages 263–270, Philadelphia. find
  16. Eisner, Jason. 1996. Three new probabilistic models for dependency parsing: An exploration. In Proceedings of the 16th International Conference on Computational Linguistics (COLING-96), pages 340–345, Copenhagen, August. find our card
  17. Gildea, Daniel. 2001. Corpus variation and parser performance. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing, Pittsburgh. find our card
  18. Gildea, Daniel and Daniel Jurafsky. 2000. Automatic labeling of semantic roles. In Proceedings of ACL 2000, Hong Kong. find our card
  19. Gildea, Daniel and Martha Palmer. 2002. The necessity of parsing for predicate argument recognition. In Proceedings of ACL 2002, Philadelphia. find
  20. Goodman, Joshua. 1997. Global thresholding and multiple-pass parsing. In Proceedings of the Second Conference on Empirical Methods in Natural Language Processing, Brown University, Providence, RI. find
  21. Henderson, John C. and Eric Brill. 1999. Exploiting diversity in natural language processing: Combining parsers. In Proceedings of the Fourth Conference on Empirical Methods in Natural Language Processing, College Park, MD. find
  22. Hwa, Rebecca. 2001. On minimizing training corpus for parser acquisition. In Proceedings of the Fifth Computational Natural Language Learning Workshop, Toulouse, France, July. find
  23. Hwa, Rebecca, Philip Resnik, and Amy Weinberg. 2002. Breaking the resource bottleneck for multilingual parsing. In Workshop on Linguistic Knowledge Acquisition and Representation: Bootstrapping Annotated Language Data, Third International Conference on Language Resources and Evaluation (LREC-2002),… find
  24. Lari, K. and S. J. Young. 1990. The estimation of stochastic context-free grammars using the Inside-Outside algorithm. Computer Speech and Language, 4:35–56. find
  25. Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313–330. find our card
  26. Ratnaparkhi, Adwait. 1996. A maximum entropy model for part-of-speech tagging. In Conference on Empirical Methods in Natural Language Processing, University of Pennsylvania, Philadelphia, May. find our card
  27. Witten, I. T. and T. C. Bell. 1991. The zero-frequency problem: Estimating the probabilities of novel events in adaptive text compression. IEEE Transactions on Information Theory 37: 1085–1094. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bikel-2004-intricacies-collins-parsing-model/