Elamur Research
☰
home / papers / 2006 / Effective Self-Training for Parsing

Effective Self-Training for Parsing

What it is about

Self-training of parsers was thought to fail because the parser's own errors are amplified. This paper shows that it works when the self-labelled trees come from a two-stage parser and reranker: the 50-best parses of 24 million unlabelled News sentences are reranked, and the reranker's best tree is added to the Wall Street Journal training data, with the two counts merged. Adding the parser's own best trees does not help, but adding the reranker's best trees lifts the first-stage parser from 90.3 to 91.0 F on section 22 at 750k added sentences, and the full system reaches 92.1 F on WSJ, a 12% error reduction. The gain levels off near 91.0 F as more sentences are added.

Ideas

For Mova

The ideas touch the silver corpus in corpus/ and the agreement rules in tools/treeagree (src/rules.rs), and the training counts of en/src/parse.rs. The chapter that matters for the teacher-student setup is languages/en/37-register-and-variation, since the school corpus is not EWT. Measure first I1 on EWT dev, with the cv folds, and keep the EWT test for one run per release.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/N06-1020.pdf

Summary

Conventional wisdom held that self-training does not help statistical parsers: when a parser is retrained on its own output, its errors tend to be reinforced. McClosky, Charniak and Johnson show this is not always true. They use the Brown two-stage system: a generative lexicalized parser (Charniak, 2000) that proposes 50-best trees, followed by a MaxEnt discriminative reranker. This system parses roughly two million LA Times sentences from the unlabeled NANC corpus. The reranker's top choice for each sentence is then added to the Penn Treebank WSJ training data, and only the first-stage parser is retrained. When the parser's own top trees are used instead, accuracy does not improve, which matches earlier reports. With reranker-selected trees, accuracy rises clearly. The final model reaches 92.1% f-score on WSJ section 23, compared with the earlier best of 91.0%. The work matters because it shows that cheap, abundant raw text can improve a parser that already performs very well. It also identifies the reranking step as the likely reason self-training works here.

Key points

  • Data and splits:
    • Labeled data: about 40k WSJ sentences from the Penn Treebank. Sections 2–21 are used for training, 24 for development, and 23 for the final test.
    • Unlabeled data: NANC, about 24M news sentences, of which only the LA Times portion is used.
  • Reranker features: The reranker uses 1,333,519 features, including 601,577 new ones based on the POS tags and words next to constituent edges. These new features alone raise the baseline from 91.0 to 91.3 on section 23.
  • Parser-best vs reranker-best (section 22, first-stage parser only):
    • Adding the parser's own top trees leaves the f-score near 90.0, with no gain.
    • Adding reranker-best trees raises it from 90.3 to about 91.0.
    • Most of the gain already appears after the first 50k added sentences.
  • Final configuration: WSJ counts weighted ×5, plus 1,750k reranker-best NANC sentences, combined by count merging. The total gain of +1.1 splits into +0.8 from self-training and +0.3 from the new features, which the authors describe as roughly a 12% error reduction.
  • Error analysis:
    • Gains concentrate in mid-length sentences, which the authors call the "Goldilocks effect".
    • Sentences with more conjunctions (CCs) also benefit.
    • There is no gain for unknown words or PP attachment.
    • Retraining the reranker on the self-trained data did not help significantly.

Interesting ideas

  • Filter self-labels through a second, stronger model. Let a model with richer features choose the pseudo-labels, then train the base model on them. Plain self-training only feeds a model its own mistakes; this approach could carry over to taggers or dependency parsers for low-resource languages.
  • Weight the gold data when mixing it with pseudo-labeled data. The authors merge event counts and give the hand-annotated data extra weight. This keeps the estimates anchored to reliable annotations while still drawing on large amounts of raw text, and higher gold weights reached a higher accuracy plateau.
  • Measure the curve as you add data. The authors evaluate after each increment of added sentences rather than at a single size. This showed that small additions give most of the benefit and that accuracy then levels off, a cheap check worth running before parsing millions of sentences.
  • Analyse errors by sentence properties. They model which sentence features predict improvement over the baseline (length, number of CCs, unknown words) and use AIC to select factors. This reveals where a method actually helps, and here it overturned the authors' expectations about unknown words.
  • Isolate each component's contribution. Evaluating the first-stage parser alone separately from the full reranking parser shows whether the gains are new information or simply duplicate what the reranker already knows. The same check applies to any multi-stage pipeline.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1220835.1220855

@inproceedings{mcclosky2006effective,
  title = {{Effective Self-Training for Parsing}},
  author = {McClosky, David and Charniak, Eugene and Johnson, Mark},
  booktitle = {Proceedings of the Human Language Technology Conference of the NAACL, Main Conference},
  year = {2006},
  doi = {10.3115/1220835.1220855},
  url = {https://doi.org/10.3115/1220835.1220855}
}

References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michiel Bacchiani, Michael Riley, Brian Roark, and Richard Sproat. 2006. MAP adaptation of stochastic grammars. Computer Speech and Language, 20(1):41–68. find
  2. Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the 11th Annual Conference on Computational Learning Theory (COLT-98). find our card
  3. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 173–180, Ann Arbor, Michigan, June. Association for Computational Linguistics. find our card
  4. Eugene Charniak. 1997. Statistical parsing with a context-free grammar and word statistics. In Proceedings of the Fourteenth National Conference on Artificial Intelligence, Menlo Park. AAAI Press/MIT Press. find
  5. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In 1st Annual Meeting of the NAACL. find
  6. Stanley F. Chen and Joshua Goodman. 1996. An empirical study of smoothing techniques for language modeling. In Arivind Joshi and Martha Palmer, editors, Proceedings of the Thirty-Fourth Annual Meeting of the Association for Computational Linguistics. find
  7. Stephen Clark, James Curran, and Miles Osborne. 2003. Bootstrapping POS-taggers using unlabelled data. In Proceedings of CoNLL-2003. find
  8. Michael Collins. 2000. Discriminative reranking for natural language parsing. In Machine Learning: Proceedings of the 17th International Conference (ICML 2000), pages 175–182, Stanford, California. find our card
  9. Sanjoy Dasgupta, M.L. Littman, and D. McAllester. 2001. PAC generalization bounds for co-training. In Advances in Neural Information Processing Systems (NIPS), 2001. find
  10. Daniel Gildea. 2001. Corpus variation and parser performance. In Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  11. David Graff. 1995. North American News Text Corpus. Linguistic Data Consortium. LDC95T21. find
  12. James Henderson. 2004. Discriminative training of a neural network statistical parser. In Proc. 42nd Meeting of Association for Computational Linguistics (ACL 2004), Barcelona, Spain. find
  13. Donald Hindle and Mats Rooth. 1993. Structural ambiguity and lexical relations. Computational Linguistics, 19(1):103–120. find
  14. Liang Huang and David Chang. 2005. Better k-best parsing. Technical Report MS-CIS-05-08, Department of Computer Science, University of Pennsylvania. find
  15. Victor M. Jimenez and Andres Marzal. 2000. Computation of the n best parse trees for weighted and stochastic context-free grammars. In Proceedings of the Joint IAPR International Workshops on Advances in Pattern Recognition. find
  16. Mark Johnson, Stuart Geman, Stephen Canon, Zhiyi Chi, and Stefan Riezler. 1999. Estimators for stochastic “unification-based” grammars. In The Proceedings of the 37th Annual Conference of the Association for Computational Linguistics, pages 535–541, San Francisco. Morgan Kaufmann. find
  17. Dan Klein and Christopher Manning. 2002. A generative constituent-context model for improved grammar induction. In Proceedings of the 40th Annual Meeting of the ACL. find
  18. Michell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  19. Anoop Sarkar. 2001. Applying cotraining methods to statistical parsing. In Proceedings of the 2001 NAACL Conference. find
  20. Mark Steedman, Miles Osborne, Anoop Sarkar, Stephen Clark, Rebecca Hwa, Julia Hockenmaier, Paul Ruhlen, Steven Baker, and Jeremiah Crim. 2003. Bootstrapping statistical parsers from small datasets. In Proceedings of EACL 03. find