Elamur Research⌕
☰
home / papers / 2001 / Improving Accuracy in word class tagging through the Combination of Machine Learning Systems

Improving Accuracy in word class tagging through the Combination of Machine Learning Systems

Paper · 2001 · Hans Van Halteren, Jakub Zavrel and Walter Daelemans

Summary

The paper tests whether different data-driven learners trained on the same NLP task make errors that differ enough for a combined system to beat the best single learner. The task is morphosyntactic word class (POS) tagging. Data-driven taggers already reach about 96–97% accuracy on this task but still do poorly on certain ambiguous words. The authors train four off-the-shelf tagger generators on identical data from three tagged corpora, one of which is LOB. The generators are a hidden Markov model, a memory-based tagger, transformation rules and maximum entropy. Their outputs are merged in two ways. Voting schemes, plain or weighted by precision and recall, produce a "gang" effect. Stacked second-level classifiers, which learn from patterns of agreement and disagreement, produce an "arbiter" effect. Cross-validation gives these second-stage learners far more training material than the authors' 1998 study had. Every combined tagger is more accurate than its strongest component, and on LOB the error rate falls by up to 24.3%. The study shows that combining heterogeneous learners can improve accuracy without new representations or extra annotated data.

Key points

  • Four component tagger generators are used: HMM, memory-based (related tools: MBT, TiMBL), transformation-based rules and maximum entropy. All are trained on the same corpus material.
  • The voting variants are Majority, TotPrecision, TagPrecision and Precision-Recall, with ties broken at random.
  • The stacking combiners use memory-based learning, maximum entropy and decision trees as second-stage learners. The paper also introduces a new method based on grouped voting.
  • The best result is a 24.3% error-rate reduction on the LOB corpus. Earlier work by the same authors (1998) had reported a 19.1% reduction for its best combiner.
  • Cross-validation was used to enlarge the second-level training data. It addresses the earlier failure to clearly confirm that arbiter-style combiners outperform voting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AntwerpAntwerp · Belgium17 works · 3,490 citations🎓Radboud University NijmegenNijmegen · Netherlands20 works · 1,688 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Hans Van Halteren, Jakub Zavrel and Walter Daelemans. 2001. Improving Accuracy in word class tagging through the Combination of Machine Learning Systems. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@article{halteren2001improving,
  title = {{Improving Accuracy in word class tagging through the Combination of Machine Learning Systems}},
  author = {Hans {Van Halteren}, Jakub Zavrel and Walter Daelemans},
  journal = {Computational Linguistics},
  year = {2001},
  doi = {10.1162/089120101750300508},
  url = {https://doi.org/10.1162/089120101750300508}
}

References

48 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abney, S., R. E. Schapire, and Y. Singer. 1999. Boosting applied to tagging and PP attachment. In Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 38-45. find
  2. Agirre, E., K. Gojenola, K. Sarasola, and A. Voutilainen. 1998. Towards a single proposal in spelling correction. In COLING-ACL "98, pages 22-28. find
  3. Alegre, M., J. Sopena, and A. Lloberas. 1999. PP-attachment: A committee machine approach. In Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 231-238. find
  4. Ali, K. M., and M. J. Pazzani. 1996. Error reduction through learning multiple descriptions. Machine Learning, 24(3):173-202. find
  5. Alpaydin, E. 1998. Techniques for combining multiple learners. In E. Alpaydin, editor, Proceedings of Engineering of Intelligent Systems, pages 6-12. find
  6. Berger, A., S. Della Pietra, and V. Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39-71.
  7. Berghmans, J. 1994. Wotan, een automatische grammatikale tagger voor het Nederlands. Master's thesis, Dept. of Language and Speech, University of Nijmegen. find
  8. Brants, T. 2000. TnT — A statistical part-of-speech tagger. In Proceedings of the Sixth Applied Natural Language Processing Conference, (ANLP-2000), pages 224-231, Seattle, WA.
  9. Breiman, L. 1996a. Bagging predictors. Machine Learning, 24(2):123-140.
  10. Breiman, L. 1996b. Stacked regressions. Machine Learning, 24(3):49-64. find
  11. Brill, E. 1992. A simple rule-based part-of-speech tagger. In Proceedings of the Third ACL Conference on Applied NLP, pages 152-155, Trento, Italy.
  12. Brill, E. 1994. Some advances in transformation-based part-of-speech tagging. In Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI '94). find
  13. Brill, E. and Jun Wu. 1998. Classifier combination for improved lexical disarnbiguation. In COLING-ACL '98: 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, pages 191-195, Montreal, Quebec, Canada. find
  14. Chan, P. K., S. J. Stolfo, and D. Wolpert. 1999. Guest editors' introduction. Special Issue on Integrating Multiple Learned Models for Improving and Scaling Machine Learning Algorithms. Machine Learning, 36(1-2):5-7. find
  15. Charniak, E. 1993. Statistical Language Learning. MIT Press, Cambridge, MA. find
  16. Chen, J., S. Bangalore, and K. Vijay-Shanker. 1999. New models for improving supertag disambiguation. In Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 188-195. find
  17. Cherkauer, K. J. 1996. Human expert-level performance on a scientific image analysis task by a system using combined artificial neural networks. In P. Chan, editor, Working Notes of the AAAI Workshop on Integrating Multiple Learned Models, pages 15-21. find
  18. Church, K. W. 1988. A stochastic parts program and noun phrase parser for unrestricted text. In Proceedings of the Second Conference on Applied Natural Language Processing.
  19. Daelemans, W., A. Van den Bosch, and A. Weijters. 1997. IGTree: Using trees for compression and classification in lazy learning algorithms. Artificial Intelligence Review, 11:407-423.
  20. Daelemans, W., J. Zavrel, P. Berck, and S. Gillis. 1996. MBT: A memory-based part of speech tagger generator. In E. Ejerhed and I. Dagan, editors, Proceedings of the Fourth Workshop on Very Large Corpora, pages 14-27. ACL SIGDAT.
  21. Daelemans, W., J. Zavrel, K. Van der Sloot, and A. Van den Bosch. 1999. TiMBL: Tilburg memory based learner, version 2.0, reference manual. Technical Report ILK-9901, ILK, Tilburg University. find
  22. Dehaspe, L. 1997. Maximum entropy modeling with clausal constraints. In Inductive Logic Programming: Proceedings of the 7th International Workshop (ILP-97), Lecture Notes in Artificial Intelligence, 1297, pages 109-124. Springer Verlag. find
  23. DeRose, S. 1988. Grammatical category disambiguation by statistical optimization. Computational Linguistics, 14:31-39.
  24. Dietterich, T. G. 1997. Machine learning research: Four current directions. AI Magazine, 18(4):97-136. find
  25. Dietterich, T. G. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895-1924. find
  26. Dietterich, T. G. and G. Bakiri. 1995. Solving multiclass learning problems via error-correcting output codes. Journal of Artificial Intelligence Research, 2:263-286. find
  27. Freund, Y. and R. E. Schapire. 1996. Experiments with a new boosting algorithm. In L. Saitta, editor, Proceedings of the 13th International Conference on Machine Learning, ICML '96, pages 148-156, San Francisco, CA. Morgan Kaufmann. find
  28. Geerts, G., W. Haeseryn, J. de Rooij, and M. van der Toorn. 1984. Algemene Nederlandse Spraakkunst. Wolters-Noordhoff, Groningen and Wolters, Leuven. find
  29. Golding, A. R. and D. Roth. 1999. A winnow-based approach to context-sensitive spelling correction. Machine Learning, 34(1-3):107-130. find
  30. Henderson, J. and E. Brill. 1999. Exploiting diversity in natural language processing: Combining parsers. In Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 187-194. find
  31. Johansson, S. 1986. The Tagged LOB Corpus: User's Manual. Norwegian Computing Centre for the Humanities, Bergen, Norway. find
  32. Marcus, M., B. Santorini, and M. A. Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
  33. Marquez, L., H. Rodrfguez, J. Carmona, and J. Montolio. 1999. Improving POS tagging using machine-learning techniques. In Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 53-62. find
  34. Quinlan, J. R. 1993. c4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA.
  35. Ratnaparkhi, A. 1996. A maximum entropy model for part-of-speech tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 133-142, University of Pennsylvania.
  36. Rigau, G., J. Atserias, and E. Agirre. 1997. Combining unsupervised lexical knowledge methods for word sense disambiguation. In Proceedings of ACL/EACL "97, pages 48-55. find
  37. Roth, D. 1998. Learning to resolve natural language ambiguities: A unified approach. In Proceedings of AAAI '98, pages 806-813. find
  38. Samuelsson, Christer. 1996. Handling sparse data by successive abstraction. In Proceedings of the 16th International Conference on Computational Linguistics, COLING-96, Copenhagen, Denmark. find
  39. Schapire, R. E. and Y. Singer. 1998. BoosTexter: A system for multiclass multi-label text categorization. Technical Report, AT&T Labs. To appear in Machine Learning. find
  40. Schapire, R. E., Y. Singer, and A. Singhal. 1998. Boosting and Rocchio applied to text filtering. In Proceedings of the 21st Annual International Conference on Research and Development in Information Retrieval, (SIGIR "98), pages 215-223. find
  41. Ting, K. M. and I. H. Witten. 1997a. Stacked generalization: When does it work? In International Joint Conference on Artificial Intelligence, Japan, pages 866-871. find
  42. Ting, K. M. and I. H. Witten. 1997b. Stacking bagged and dagged models. In International Conference on Machine Learning, Tennessee, pages 367-375. find
  43. Tufi~, D. 1999. Tiered tagging and combined language models classifiers. In Proceedings Workshop on Text, Speech, and Dialogue. find
  44. Uit den Boogaart, P. C. 1975. find
  45. Woordfrequenties in geschreven en gesproken Nederlands. Utrecht, Oosthoek, Scheltema & Holkema. van Halteren, H. 1996. Comparison of tagging strategies, a prelude to democratic tagging. In S. Hockey and N. Ide, editors, Research in Humanities Computing 4. Selected Papers from the ALLC/ACH Conference… find
  46. Syntactic Wordclass Tagging. Kluwer Academic Publishers, Dordrecht, The Netherlands. van Halteren, H. 2000a. A default first order weight determination procedure for WPDV models. In Proceedings of CoNLL-2000 and LLL-2000, pages 119-122, Lisbon, Portugal. van Halteren, H. 2000b. Chunking with WPDV mo… find
  47. Wolpert, D. H. 1992. Stacked generalization. Neural Networks, 5:241-259. find
  48. Zavrel, J. and W. Daelemans. 2000. Bootstrapping a tagged corpus through combination of existing heterogeneous taggers. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC 2000), pages 17-20. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/halteren-2001-improving-accuracy-word-class-tagging/