Elamur Research⌕
☰
home / papers / 2001 / Scaling to Very Very Large Corpora for Natural Language Disambiguation

Scaling to Very Very Large Corpora for Natural Language Disambiguation

Summary

Banko and Brill ask whether conclusions drawn from training NLP learners on roughly a million words still hold when far more data is available. They use confusion set disambiguation (choosing between words such as {then, than} or {weather, whether}) as a test case, because correct labels come for free from well-edited text. They built a 1-billion-word training corpus from news, scientific abstracts, government transcripts, literature and other prose, and evaluated on a held-out set of 1 million words drawn from the Wall Street Journal. Four learners were trained on growing slices of this corpus: winnow, perceptron, naïve Bayes and a simple memory-based learner. Accuracy for every learner kept rising roughly log-linearly up to the full billion words, with no sign of levelling off. At the same time, model size grew and the benefit of voting disappeared. For settings where labels are costly, the authors study committee-based active learning and self-labeling with bagged classifiers. The work argues that the field should spend more effort building larger annotated collections and less on comparing algorithms trained on small data.

Key points

  • Training data reached 10⁹ words, about three orders of magnitude more than earlier work on this task. Results are averaged over ten confusion sets.
  • Learning curves for winnow, perceptron, naïve Bayes and the memory-based learner stayed log-linear up to one billion words. The memory-based learner used only the neighbouring words as features.
  • Complementarity between two learners fell from 0.2612 at 10⁶ words to 0.1612 at 10⁹ words. Voting helped only with very small corpora and slightly reduced accuracy at the largest sizes.
  • With 10 bagged naïve Bayes models trained on a 1M-word seed corpus, unanimous agreement gave 0.8734 accuracy, while a 5–5 split gave 0.5000.
  • Self-training on unanimously labeled instances helped only up to a point, then accuracy declined:
    • {then, than}: from 0.9624 to a peak of 0.9715, versus 0.9878 with fully supervised 10⁹ words.
    • {among, between}: from 0.8183 to 0.8335, versus 0.9021 fully supervised.
    • Adding all automatically labeled instances, instead of only the most certain ones, could sharply hurt accuracy (0.7604 vs 0.9588 at 5×10⁸ words for {then, than}).

Interesting ideas

  • I1. Check the data curve first. Before spending effort on new algorithms or features, plot learning curves over increasing data sizes. This tells you whether collecting or annotating more text is the cheaper path to gains, which applies to any low-resource language project.
  • I2. Find tasks with free labels. Some tasks have labels hidden in ordinary text, as confusable words do in edited prose. Such tasks are cheap testbeds for scaling studies and good sources of large training sets.
  • I3. Sample sources proportionally. When building ever-larger training slices from mixed sources, draw sentences from each source in proportion to its size instead of concatenating them. This keeps each slice representative and avoids order-related bias.
  • I4. Use one committee for two purposes. A bagged committee of classifiers can both pick uncertain examples for human annotation and pick high-agreement examples for automatic labeling. It is a simple, reusable tool for stretching a limited annotation budget.
  • I5. Re-test ensembles at scale. Voting gains measured on small data may vanish or reverse with large data, so ensemble benefits should be re-checked at realistic data sizes. Self-labeling should likewise be monitored for the point where its sample bias starts to outweigh the extra data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

110 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 102 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1073012.1073017

@inproceedings{banko2001scaling,
  title = {{Scaling to Very Very Large Corpora for Natural Language Disambiguation}},
  author = {Banko, Michele and Brill, Eric},
  booktitle = {Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics},
  year = {2001},
  doi = {10.3115/1073012.1073017},
  url = {https://doi.org/10.3115/1073012.1073017}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Banko, M. and Brill, E. (2001). Mitigating the Paucity of Data Problem. Human Language Technology. find
  2. Breiman L., (1996). Bagging Predictors, Machine Learning 24 123-140. find
  3. Brill, E. and Wu, J. (1998). Classifier combination for improved lexical disambiguation. In Proceedings of the 17th International Conference on Computational Linguistics. find
  4. Charniak, E. (1996). Treebank Grammars, Proceedings AAAI-96 , Menlo Park, Ca. find
  5. Dagan, I. and Engelson, S. (1995). Committee-based sampling for training probabilistic classifiers. In Proc. ML-95, the 12th Int. Conf. on Machine Learning. find
  6. Gale, W. A., Church, K. W., and Yarowsky, D. (1993). A method for disambiguating word senses in a large corpus. Computers and the Humanities, 26:415--439. find
  7. Golding, A. R. (1995). A Bayesian hybrid method for context-sensitive spelling correction. In Proc. 3rd Workshop on Very Large Corpora, Boston, MA. find
  8. Golding, A. R. and Roth, D.(1999), A Winnow-Based Approach to Context-Sensitive Spelling Correction. Machine Learning, 34:107--130. find
  9. Golding, A. R. and Schabes, Y. (1996). Combining trigram-based and feature-based methods for context-sensitive spelling correction. In Proc. 34th Annual Meeting of the Association for Computational Linguistics, Santa Cruz, CA. find
  10. Henderson, J. C. and Brill, E. (1999). Exploiting diversity in natural language processing: combining parsers. In 1999 Joint Sigdat Conference on Empirical Methods in Natural Language Processing and Very Large Corpora. ACL, New Brunswick NJ. 187-194. find
  11. Jones, M. P. and Martin, J. H. (1997). Contextual spelling correction using latent semantic analysis. find
  12. In Proc. 5th Conference on Applied Natural Language Processing, Washington, DC. find
  13. Lewis , D. D., & Catlett, J. (1994). Heterogeneous uncertainty sampling. Proceedings of the Eleventh International Conference on Machine Learning (pp. 148--156). New Brunswick, NJ: Morgan Kaufmann. find
  14. Mangu, L. and Brill, E. (1997). Automatic rule acquisition for spelling correction. In Proc. 14th International Conference on Machine Learning. Morgan Kaufmann. find
  15. Merialdo, B. (1994). Tagging English text with a probabilistic model. Computational Linguistics, 20(2):155--172. find our card
  16. Mitchell, T. M. (1999), The role of unlabeled data in supervised learning, in Proceedings of the Sixth International Colloquium on Cognitive Science, San Sebastian, Spain. find
  17. Nigam, N., McCallum, A., Thrun, S., and Mitchell, T. (1998). Learning to classify text from labeled and unlabeled documents. In Proceedings of the Fifteenth National Conference on Artificial Intelligence. AAAI Press.. find
  18. Pedersen, T. (2000). A simple approach to building ensembles of naive bayesian classifiers for word sense disambiguation. In Proceedings of the First Meeting of the North American Chapter of the Association for Computational Linguistics May 1-3, 2000, Seattle, WA Powers, D. (1997). Learning and appl… find
  19. Weng, F., Stolcke, A, & Sankar, A (1998). Efficient lattice representation and generation. Proc. Intl. Conf. on Spoken Language Processing, vol. 6, pp. 2531-2534. Sydney, Australia. find
  20. Yarowsky, D. (1994). Decision lists for lexical ambiguity resolution: Application to accent restoration in Spanish and French. In Proc. 32nd Annual Meeting of the Association for Computational Linguistics, Las Cruces, NM. find our card
  21. Yarowsky, D. (1995) Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics. Cambridge, MA, pp. 189-196, 1995. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/banko-2001-scaling-very-very-large-corpora/