Elamur Research⌕
☰
home / papers / 2018 / Sprucing up the trees – Error detection in treebanks

Sprucing up the trees – Error detection in treebanks

Paper · 2018 · Ines Rehbein and Josef Ruppenhofer

Summary

Most UD treebanks were converted automatically from other annotation schemes, so they carry conversion noise on top of ordinary human inconsistencies, and full manual review is usually too expensive. The authors extend MACE-AL, an earlier error detector for token-level tasks such as POS tagging and NER, to dependency trees. The new system, MACE-AL-TREE, takes the output of a committee of five dependency parsers and fits two separate variational inference models, one for edge labels and one for head attachments, to estimate how reliable each parser is. Tokens with the highest posterior entropy go to a human, or to the gold tree in simulation, and the correction is fed back into the model. Final trees are built either from the local decisions directly or with Chu-Liu-Edmonds over competence-weighted votes. Error-detection precision ranged from 35% to above 90%. Without any human input, the method produced self-training data that raised out-of-domain LAS by up to about 2%. This matters because it offers a cheap way to clean treebanks and adapt parsers to new domains.

Key points

  • MACE-AL-TREE builds on MACE (Hovy et al., 2013), which models annotators as either answering truthfully or "spamming," and on MACE-AL, which adds active-learning feedback. Without that feedback, plain MACE could not beat majority vote on machine-generated labels.

  • Label errors and attachment errors are handled by two distinct variational models. A single joint model would have too large a label space and would train too slowly for interactive use.

  • The committee contains five parsers, including the neural BiLSTM graph parser (BIST Graph) and the RBG parser. The parsers are trained only once, before active learning begins, so the correction loop never has to retrain them.

  • Two ways of producing output trees are compared: MACE-AL-TREE takes the most trusted local labels and heads, which can leave trees disconnected, and MACE-AL-TREE-CLE applies Chu-Liu-Edmonds to competence-weighted votes.

  • In the web-genre domain adaptation experiments, self-training on MACE-AL-TREE output (up to 25,000 added sentences) beat two baselines:

    • self-training on the best single parser's output;
    • keeping only trees where two parsers agree, a filter that tends to favour short sentences.

    Gains over training on the original data alone appeared only for emails and reviews, the genres with the most unknown words.

Interesting ideas

  • I1. Treat each parser as an annotator with a learned reliability, then rank tokens by posterior entropy so that review effort goes where the committee is most likely wrong.
  • I2. Split one structured decision into separate subproblems, here labels and heads, each with its own small inference model, so the label space stays manageable and retraining stays fast.
  • I3. Reconcile local per-token decisions into a well-formed tree by running Chu-Liu-Edmonds over a vote matrix weighted by estimated competence.
  • I4. Fold each correction back in by overwriting one randomly chosen committee member's prediction, then re-estimate trust, without retraining the expensive base models.
  • I5. Agreement-only filtering of self-training data quietly skews the data toward short, easy sentences.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data agreed by several witness parsers could be weighted by learned per-parser competence rather than raw voting, and the highest-entropy tokens could be routed to the LLM teacher tier first.
  • I2 (tree): When fusing gparse and perceptron parser outputs, label and attachment trust could be modelled separately, since a parser may be good at heads but weak on relation labels.
  • I3 (tree): An ensemble of witness parsers could always yield a valid UD tree, which downstream frame extraction requires, instead of a possibly disconnected majority patchwork.
  • I4 (learn): Self-induced repair rules or teacher corrections could update the agreement model cheaply between full training runs, keeping the correction loop interactive.
  • I5 (learn): Silver-data selection should be checked for sentence-length and construction bias, for example with a negative control comparing distributions against the raw pool.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany102 works · 17,447 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Treebanks · 118 works

Cite

Cite as Ines Rehbein and Josef Ruppenhofer. 2018. Sprucing up the trees – Error detection in treebanks. Proceedings of the 27th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{rehbein2018sprucing,
  title = {{Sprucing up the trees – Error detection in treebanks}},
  author = {Ines Rehbein and Josef Ruppenhofer},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  url = {https://aclanthology.org/C18-1010/}
}

References

32 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bhasha Agrawal, Rahul Agarwal, Samar Husain, and Dipti M. Sharma. 2013. An automatic approach to treebank error detection using a dependency parser. In Proceedings of the 14th International Conference on Computational Linguistics and Intelligent Text Processing - Volume Part I, CICLing’13, pages 294–303. find
  2. Ann et al. Bies. 2012. English Web Treebank LDC2012T13. find
  3. Anders Björkelund and Joakim Nivre. 2015. Non-deterministic oracles for unrestricted non-projective transition-based dependency parsing. In Proceedings of the 14th International Conference on Parsing Technologies, pages 76–86, Bilbao, Spain, July. Association for Computational Linguistics. find
  4. Adriane Boyd, Markus Dickinson, and Detmar Meurers. 2008. On detecting errors in dependency treebanks. Research on Language and Computation, 6:113–137. ↗
  5. Sabine Brants, Stefanie Dipper, Silvia Hansen, Wolfgang Lezius, and George Smith. 2002. The TIGER treebank. In Proceedings of the Workshop on Treebanks and Linguistic Theories, TLT’02, pages 24–41. find
  6. Yoeng-Jin Chu and Tseng-Hong Liu. 1965. On shortest arborescence of a directed graph. Scientia Sinica, 14(10):1396–1400. find
  7. Markus Dickinson and Detmar Meurers. 2003. Detecting inconsistencies in treebanks. In The Second Workshop on Treebanks and Linguistic Theories, TLT. find
  8. Markus Dickinson and W. Detmar Meurers. 2005. Prune Diseased Branches to Get Healthy Trees! How to Find Erroneous Local Trees in a Treebank and Why It Matters. In Proceedings of the Fourth Workshop on Treebanks and Linguistic Theories, TLT 2005, pages 41–52, Barcelona, Spain. find
  9. Markus Dickinson and Amber Smith. 2011. Detecting dependency parse errors with minimal resources. In Proceedings of the 12th International Conference on Parsing Technologies, IWPT, pages 241–252. find
  10. Markus Dickinson and Amber Smith. 2017. Simulating dependencies to improve parse error detection. In Proceedings of the 15th International Workshop on Treebanks and Linguistic Theories, TLT15, pages 76–88. find
  11. Jack Edmonds. 1967. Optimum branchings. Journal of Research of the National Bureau of Standards B, 71(4):233–240. find
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computing, 9(8):1735–1780, November. ↗
  13. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT’13, pages 1120–1130, Atlanta, Georgia, USA.
  14. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics (TACL), 4:313– 327. ↗
  15. Jonathan K. Kummerfeld, David Hall, James R. Curran, and Dan Klein. 2012. Parser showdown at the Wall Street Corral: An empirical investigation of error types in parser output. In The 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’12, pages 1048–1059.
  16. Tao Lei, Yu Xin, Yuan Zhang, Regina Barzilay, and Tommi S. Jaakkola. 2014. Low-rank tensors for scoring dependency structures. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long Papers, pages 1381–1391. find
  17. Ryan McDonald and Joakim Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL’2007, pages 122–131.
  18. Jens Nilsson and Joakim Nivre. 2008. MaltEval: An evaluation and visualization tool for dependency parsing. In Proceedings of the Sixth International Conference on Language Resources and Evaluation, LREC’08, Marrakech, Morocco, may. European Language Resources Association (ELRA). find
  19. Joakim Nivre, Johan Hall, and Jens Nilsson. 2006. MaltParser: A data-driven parser-generator for dependency parsing. In Linguistic Resources and Evaluation, LREC’2006, pages 2216–2219.
  20. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may. European Language Resources Association (ELRA).
  21. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). find
  22. Sujith Ravi, Kevin Knight, and Radu Soricut. 2008. Automatic prediction of parser accuracy. In Empirical Methods in Natural Language Processing, EMNLP’08, pages 887–1575. find
  23. Ines Rehbein and Josef Ruppenhofer. 2017. Detecting annotation noise in automatically labelled data. In Proceedings of the 55th annual meeting of the Association for Computational Linguistics, ACL’17, Vancouver, Canada. find
  24. Roi Reichart and Ari Rappoport. 2007. Self-training for enhancement and domain adaptation of statistical parsers trained on small datasets. In The 45th Annual Meeting of the Association of Computational Linguistics, pages 616–623. find
  25. Djamé Seddah, Sandra Kübler, and Reut Tsarfaty. 2014. Introducing the SPMRL 2014 shared task on parsing morphologically-rich languages. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages, pages 103–109, Dublin, Ireland, August. Dublin City University. find
  26. Burr Settles. 2009. Active learning literature survey. Wisconsin–Madison. Computer sciences technical report, University of Anders Søgaard and Christian Rishøj. 2010. Semi-supervised dependency parsing using generalized tri-training. In Proceedings of the 23rd International Conference on Computational Linguistics, COLING ’10, pages 1065– 1073.
  27. Milan Straka and Jana Straková. 2017. Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 88–99, Stroudsburg, PA, USA. Association for Computational Linguistics.
  28. Mihai Surdeanu and Christopher D. Manning. 2010. Ensemble models for dependency parsing: Cheap and good? In Proceedings of the North American Chapter of the Association for Computational Linguistics Conference (NAACL-2010), Los Angeles, CA, June. find
  29. Tylman Ule and Kiril Simov. 2004. Unexpected productions may well be errors. In The Fourth International Conference on Language Resources and Evaluation, LREC’04. find
  30. Gertjan van Noord. 2004. Error mining for wide-coverage grammar engineering. In The 42nd Annual Meeting of the Association for Computational Linguistics, pages 446–453. find
  31. Alexander Volokh and Günter Neumann. 2011. Automatic detection and correction of errors in dependency treebanks. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 346–350.
  32. Xingxing Zhang, Jianpeng Cheng, and Mirella Lapata. 2017. Dependency parsing as head selection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL’17, pages 665–676. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rehbein-2018-sprucing-up-trees-error-detection/