Elamur Research⌕
☰
home / papers / 2007 / Single Malt or Blended? A Study in Multilingual Parser Optimization

Single Malt or Blended? A Study in Multilingual Parser Optimization

Paper · 2007 · Johan Hall, Jens Nilsson, Joakim Nivre et al.

Summary

This paper describes how the Växjö/Uppsala team adapted MaltParser to the ten languages of the CoNLL 2007 multilingual dependency parsing shared task: Arabic, Basque, Catalan, Chinese, Czech, English, Greek, Hungarian, Italian and Turkish. The work proceeded in two stages.

First, they built "Single Malt", one deterministic, SVM-driven transition parser per language. They began from a shared baseline, fixed the parsing-algorithm options, and then alternated between feature selection and learning-parameter tuning, always using LAS as the target.

Second, they built "Blended", which merges the output of six variants of the tuned parser. Following Sagae and Lavie (2006), each candidate arc gets a weighted vote, with weights set per parser and per coarse part of speech.

On the official test data, Blended beat Single Malt for every language and reached the best average LAS in the task. This shows that careful tuning of a simple deterministic parser, plus a cheap ensemble on top, produces strong results across many languages. The authors also found that languages with rich morphology remain harder to parse.

Key points

  • Development gains for Single Malt: average LAS rose from 77.78 (baseline) to 78.10 after parsing-algorithm tuning, and to 79.81 after feature and learner tuning. Feature selection gave the largest improvement. The final models had between 18 (Turkish) and 56 (Hungarian) features.
  • Test results: average LAS was 79.80 for Single Malt and 81.20 for Blended. The 81.20 figure uses a corrected Chinese run; the official average was 80.32, still the top reported score. Blended improved on Single Malt by 1.40 points on average, ranging from +0.44 (Greek) to +2.40 (English).
  • Pseudo-projective parsing: it helped only for Basque, Czech, Greek and Turkish. In these treebanks more than 20% of sentences contain non-projective structure, compared with 10% or less elsewhere.
  • Effect of language type: languages with rich morphology and flexible word order scored below 80% LAS. Configurational languages such as Catalan, Chinese, English and Italian scored above 80%. Training-set size mattered less: Italian, with little data, outperformed Czech, with a large treebank.
  • Blended best results: Blended had the highest LAS of all systems for Arabic, Basque, Catalan, Hungarian and Italian.

Interesting ideas

  • I1. Staged tuning: fix a common baseline first, settle the structural algorithm options next, and only then alternate between feature and hyperparameter search. This keeps a large configuration space manageable and makes the contribution of each stage easy to report.
  • I2. Data-dependent preprocessing: whether to apply a transformation like pseudo-projectivization should depend on a measurable property of the data, here the share of non-projective sentences. Checking corpus statistics before adopting a technique avoids wasted effort, for example on new Ukrainian UD treebanks.
  • I3. Ensembles from extrapolated variants: the component parsers were derived from settings already tuned for the single system rather than tuned from scratch. Even so, the ensemble reduced error propagation, especially on long dependencies. This is a low-cost route to an ensemble when development time is short.
  • I4. Fine-grained vote weights: each parser's vote was weighted by its accuracy for a given coarse POS tag. This lets each component contribute where it is strongest, and the idea carries over to any tagger or parser combination. The weights must be set on data kept separate from the final evaluation set.
  • I5. Targeted error analysis: the authors split precision and recall by dependency length and treated root attachments as a separate case. This showed what the ensemble fixed (long dependencies) and what it made worse (fragmented parses with multiple ROOT dependents), which an overall LAS number would hide.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Johan Hall, Jens Nilsson, Joakim Nivre, Gülşen Eryiǧit, Beáta Megyesi, Mattias Nilsson and Markus Saers. 2007. Single Malt or Blended? A Study in Multilingual Parser Optimization. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hall2007single,
  title = {{Single Malt or Blended? A Study in Multilingual Parser Optimization}},
  author = {Johan Hall, Jens Nilsson, Joakim Nivre, Gülşen Eryiǧit, Beáta Megyesi, Mattias Nilsson and Markus Saers},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1097/}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Abeillé, editor. 2003. Treebanks: Building and Using Parsed Corpora. Kluwer. find
  2. I. Aduriz, M. J. Aranzabe, J. M. Arriola, A. Atutxa, A. Diaz de Ilarraza, A. Garmendia, and M. Oronoz. 2003. Construction of a Basque dependency treebank. In Proc. of the 2nd Workshop on Treebanks and Linguistic Theories (TLT), pages 201–204. find
  3. A. Böhmová, J. Hajič, E. Hajičová, and B. Hladká. 2003. The PDT: a 3-level annotation scenario. In Abeillé (2003), chapter 7, pages 103–127. find
  4. C.-C. Chang and C.-J. Lin, 2001. LIBSVM: A Library for Support Vector Machines. Software available at http://www.csie.ntu.edu.tw/ cjlin/libsvm. ↗
  5. K. Chen, C. Luo, M. Chang, F. Chen, C. Chen, C. Huang, and Z. Gao. 2003. Sinica treebank: Design criteria, representational issues and implementation. In Abeillé (2003), chapter 13, pages 231–248. find
  6. M. A. Covington. 2001. A fundamental algorithm for dependency parsing. In Proc. of the 39th Annual ACM Southeast Conf., pages 95–102. find
  7. D. Csendes, J. Csirik, T. Gyimóthy, and A. Kocsor. 2005. The Szeged Treebank. Springer. find
  8. J. Hajič, O. Smrž, P. Zemánek, J. Šnaidauf, and E. Beška. 2004. Prague Arabic dependency treebank: Development in data and tools. In Proc. of the NEMLAR Intern. Conf. on Arabic Language Resources and Tools, pages 110–117. find
  9. R. Johansson and P. Nugues. 2007. Extended constituent-to-dependency conversion for English. In Proc. of the 16th Nordic Conference on Computational Linguistics (NODALIDA).
  10. M. Marcus, B. Santorini, and M. Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330.
  11. M. A. Martı́, M. Taulé, L. Màrquez, and M. Bertran. 2007. CESS-ECE: A multilingual and multilevel annotated corpus. Available for download from: http://www.lsi.upc.edu/∼mbertran/cess-ece/. ↗
  12. R. McDonald and J. Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In Proc. of the Joint Conf. on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).
  13. R. McDonald, F. Pereira, K. Ribarov, and J. Hajič. 2005. Non-projective dependency parsing using spanning tree algorithms. In Proc. of the Human Language Technology Conf. and the Conf. on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 523–530.
  14. S. Montemagni, F. Barsotti, M. Battista, N. Calzolari, O. Corazzari, A. Lenci, A. Zampolli, F. Fanciulli, M. Massetani, R. Raffaelli, R. Basili, M. T. Pazienza, D. Saracino, F. Zanzotto, N. Nana, F. Pianesi, and R. Delmonte. 2003. Building the Italian Syntactic-Semantic Treebank. In Abeillé (2003),… find
  15. J. Nivre and J. Nilsson. 2005. Pseudo-projective dependency. In Proc. of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 99–106. find
  16. J. Nivre, J. Hall, J. Nilsson, G. Eryiğit, and S. Marinov. 2006. Labeled pseudo-projective dependency parsing with support vector machines. In Proc. of the Tenth Conf. on Computational Natural Language Learning (CoNLL), pages 221–225.
  17. J. Nivre, J. Hall, S. Kübler, R. McDonald, J. Nilsson, S. Riedel, and D. Yuret. 2007. find
  18. The CoNLL 2007 shared task on dependency parsing. In Proc. of the Joint Conf. on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).
  19. J. Nivre. 2007. Incremental non-projective dependency parsing. In Human Language Technologies: The Annual Conf. of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 396–403. find
  20. K. Oflazer, B. Say, D. Zeynep Hakkani-Tür, and G. Tür. 2003. Building a Turkish treebank. In Abeillé (2003), chapter 15, pages 261–277. find
  21. P. Prokopidis, E. Desypri, M. Koutsombogera, H. Papageorgiou, and S. Piperidis. 2005. Theoretical and practical issues in the construction of a Greek dependency treebank. In Proc. of the 4th Workshop on Treebanks and Linguistic Theories (TLT), pages 149–160. find
  22. K. Sagae and A. Lavie. 2006. Parser combination by reparsing. In Proc. of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 129–132.
  23. H. Yamada and Y. Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proc. 8th International Workshop on Parsing Technologies (IWPT), pages 195–206.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hall-2007-single-malt-or-blended-study/