Elamur Research
☰
home / papers / 2006 / Reranking and Self-Training for Parser Adaptation

Reranking and Self-Training for Parser Adaptation

What it is about

Parsers trained on the Wall Street Journal lose much accuracy on other genres: the standard parser scores 89.7% on WSJ but only 82.9% on the Brown corpus. The authors show two techniques that raise Brown accuracy without any labeled Brown data: a reranker over the 50-best parses, which lifts Brown to 85.2%, and self-training on unlabeled out-of-domain text (NANC), where the parser's own reranked parses become new training data, which lifts it to 87.8%. That is a 28% error reduction, and it beats a parser and reranker trained on labeled Brown data (88.4%). The gains come from using unlabeled text from a different domain together with the reranker.

Ideas

For Mova

The self-training loop belongs to the training commands in en and the reranker would sit beside en/src/parse.rs. The silver corpus is the school corpus under corpus/. The chapter that the ideas touch is languages/en/08-national-varieties.md and languages/en/37-register-and-variation.md. Measure first: EWT dev LAS after one self-training round on the school corpus, compared with the current model.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P06-1043.pdf

Summary

The paper tackles parser portability: statistical parsers trained on the WSJ section of the Penn Treebank lose much accuracy on other genres, and building new treebanks for each domain is costly. The authors adapt to the Brown corpus without any labeled Brown data. They combine two techniques. The first is the Charniak and Johnson (2005) reranking parser, which reorders 50-best lists using rich features. The second is self-training on unlabeled NANC news, specifically the LA Times portion. In that step, the WSJ-trained reranking parser labels NANC sentences, and the top reranked trees are added to WSJ to retrain the first-stage parser. On the Brown test set, the plain Charniak parser scores 82.9% f-measure. Reranking lifts it to 85.2%, and self-training lifts it further to 87.8%. A parser and reranker trained entirely on labeled Brown data reaches 88.4%, so the gap is small. The result matters because it shows that richer features and self-training on news are not overfitted to WSJ and can transfer to a different genre.

Key points

  • Corpora:
    • WSJ has about 40,000 sentences. Sections 2–21 are used for training, 24 for development and 23 for testing.
    • The treebanked Brown sections have about 25,000 sentences (458,000 words), split roughly 80/10/10. They are mostly fiction.
    • NANC has about 24 million unlabeled sentences, of which only the LA Times portion is used.
  • Self-training results on Brown development:
    • Adding 2,500k self-trained NANC sentences raises the parser from 83.9% to 86.4%.
    • The same data raises the reranking parser from 85.8% to 87.7%.
    • Gains level off as more NANC data is added, and the reranking and self-training improvements appear to be orthogonal.
  • Self-training on unlabeled Brown text gives mixed results:
    • It helps the first-stage parser, which goes from 84.8% to 85.6%, and adding NANC brings it to 86.3%.
    • It hurts the reranking parser, which drops from 86.6% to 86.1%. The authors cannot explain this.
  • Significance testing:
    • A randomization test with 10^6 samples finds the Brown-trained reranker not significantly different from the WSJ-trained one (difference 0.002, p = 0.27).
    • At the sentence level, a Wilcoxon test (p = 0.0003) shows the Brown/Brown system still differs from WSJ+NANC/WSJ.
  • Switchboard: the best WSJ+NANC system gains only 3% absolute over a 74.0% baseline, an 11% error reduction. The reported error reduction on Brown is given as 28% early in the paper and 24% in the conclusions.

Interesting ideas

  • Self-train on unlabeled text from a related, nearby domain (news) rather than the target domain itself. The paper found this to be more robust than self-training on raw target text. It is a cheap option when target-domain raw text is scarce or noisy.
  • Use the reranked 1-best output to retrain only the first-stage generator, and keep the original reranker. Retraining the reranker gave no clear gain. This keeps the pipeline simple and suggests where pseudo-labels actually help.
  • Diagnose the gap between two systems with a logistic regression over per-sentence wins. The predictors are sentence features such as preposition count and subgenre ID. This reveals where each model is stronger, for example out-of-domain models doing better on preposition-heavy sentences, which corpus-level scores hide.
  • Pair corpus-level randomization tests with sentence-level paired tests. The paper shows that two systems can look equivalent at the corpus level yet differ significantly sentence by sentence, which is a useful check for any evaluation.
  • Use "self-trained bridging corpora" for distant domains, the authors' own speculative proposal. The idea is to self-train on intermediate text, such as a biology textbook before medical text, to step gradually toward a hard target domain where no treebank exists.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1220175.1220218

@inproceedings{mcclosky2006reranking,
  title = {{Reranking and Self-Training for Parser Adaptation}},
  author = {McClosky, David and Charniak, Eugene and Johnson, Mark},
  booktitle = {Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics},
  year = {2006},
  doi = {10.3115/1220175.1220218},
  url = {https://doi.org/10.3115/1220175.1220218}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michiel Bacchiani, Michael Riley, Brian Roark, and Richard Sproat. 2006. MAP adaptation of stochastic grammars. Computer Speech and Language, 20(1):41–68. find
  2. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proc. of the 2005 Meeting of the Assoc. for Computational Linguistics (ACL), pages 173–180. find our card
  3. Andrew B. Clegg and Adrian Shepherd. 2005. Evaluating and integrating treebank parsers on a biomedical corpus. In Proceedings of the ACL Workshop on Software. find
  4. Paul R. Cohen. 1995. Empirical Methods for Artificial Intelligence. The MIT Press, Cambridge, Massachusetts. find
  5. B ROWN / WSJ 0.030 (0) 0.004 (0.1) B ROWN /B ROWN 0.031 (0) 0.006 (0.025) 0.002 (0.27) find
  6. Michael Collins. 2000. Discriminative reranking for natural language parsing. In Machine Learning: Proceedings of the Seventeenth International Conference (ICML 2000), pages 175–182, Stanford, California. find our card
  7. W. Nelson Francis and Henry Kučera. 1979. Manual of Information to accompany a Standard Corpus of Present-day Edited American English, for use with Digital Computers. Brown University, Providence, Rhode Island. find
  8. Daniel Gildea. 2001. Corpus variation and parser performance. In Empirical Methods in Natural Language Processing (EMNLP), pages 167–202. find our card
  9. David Graff. 1995. North American News Text Corpus. Linguistic Data Consortium. LDC95T21. find
  10. Rebecca Hwa. 1999. Supervised grammar induction using training data with limited constituent information. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 72–80, University of Maryland. find
  11. Matthew Lease and Eugene Charniak. 2005. Parsing biomedical literature. In Second International Joint Conference on Natural Language Processing (IJC-NLP’05). find
  12. Michell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Comp. Linguistics, 19(2):313–330. find our card
  13. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of HLT-NAACL 2006. find our card
  14. Adwait Ratnaparkhi. 1999. Learning to parse natural language with maximum entropy models. Machine Learning, 34(1-3):151–175. find
  15. Satoshi Sekine. 1997. The domain dependence of parsing. In Proc. Applied Natural Language Processing (ANLP), pages 96–102. find
  16. Mark Steedman, Miles Osborne, Anoop Sarkar, Stephen Clark, Rebecca Hwa, Julia Hockenmaier, Paul Ruhlen, Steven Baker, and Jeremiah Crim. 2003. Bootstrapping statistical parsers from small datasets. In Proc. of European ACL (EACL), pages 331–338. find