Elamur Research⌕
☰
home / papers / 2010 / Improved Statistical Measures to Assess Natural Language Parser Performance across Domains

Improved Statistical Measures to Assess Natural Language Parser Performance across Domains

Paper · 2010 · Barbara Plank

Summary

Parsers trained on one kind of text often lose accuracy on another, and it is hard to tell in advance which systems will suffer most. Plank asks whether cheap, annotation-free statistics of a text can show how well a Dutch dependency parser will do on it. She compares Alpino, a hand-written HPSG-style grammar with a Maximum Entropy disambiguator, against two language-independent data-driven parsers, MST (graph-based) and Malt (transition-based). All three are trained on the cdb newspaper treebank and scored with LAS on 95 Dutch Wikipedia articles from LASSY, grouped into ten topical subdomains.

Building on Zhang and Wang's (2009) English study, she tests average sentence length, unknown word rate and unknown PoS trigram rate, and adds trigram language-model perplexity. Each measure is correlated with per-article accuracy. Perplexity turns out to be the strongest signal for every parser, and the data-driven systems are the most sensitive to it. This points toward predicting parser difficulty, and thus domain distance, before any annotation exists.

Key points

  • Data.
    • Training set: cdb, 7,136 sentences from the Eindhoven newspaper corpus.
    • Test set: 95 LASSY Wikipedia articles, 6,710 sentences and about 90k words, mostly on Belgian topics, in ten subdomains (e.g. LOC, POL, SPO).
  • Sanity check on the CoNLL 2006 Dutch test set.
    • The treebank was converted to CoNLL format and retagged with Alpino's finer PoS tags.
    • MST reached 82.14 LAS and Malt 80.64, above the earlier reported 79.19 and 78.59.
  • Outlier articles.
    • Three sports articles about bike races consist mostly of winner lists, with an average of 6.03 words per sentence versus 13.68 overall.
    • On these articles MST hit its lowest scores, while Alpino and Malt stayed strong. They were removed from the main analysis.
  • Correlations with accuracy.
    • Sentence length and UPTR: no correlation for any parser.
    • Unknown word rate: correlates for the data-driven parsers (−0.39 and −0.28) but barely for Alpino (sUWR −0.07).
    • Perplexity: the strongest measure, at −0.67 and −0.57 for the data-driven parsers and −0.33 for Alpino. The same pattern held on DPC.
  • Overall accuracy.
    • Alpino scored roughly 80–100% LAS per article.
    • MST and Malt ranged from about 50% to 95%.

Interesting ideas

  • I1. Train a word-trigram language model on the parser's training corpus and use the perplexity of new text as a cheap, annotation-free estimate of how hard that text will be to parse.
  • I2. Plotting accuracy per document against simple text statistics exposes odd genres, such as list-heavy pages, that skew the averages and behave very differently across parsers.
  • I3. Converting every system's output to one format and scoring all of them with the same LAS script makes cross-system comparison fair, provided you note what the conversion drops (secondary edges, multiword tokens).
  • I4. Retagging training data with fine-grained tags from a stronger, language-specific analyzer lifted the accuracy of generic data-driven parsers.
  • I5. Dedicated unknown-word heuristics for numbers, compounds and proper names made the grammar-based parser largely immune to vocabulary shift.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Compute this score for each input document against our treebanks to flag texts likely to be misparsed, and route them to gparse or extra scrutiny.
  • I2 (learn): Before accepting silver data, scan documents for extreme statistics such as very short "sentences" or high OOV rates, and quarantine list-like or tabular material.
  • I3 (tree): Score the perceptron parser and gparse with one UD evaluation, after conversion, and log which dialect features each conversion discards.
  • I4 (tokens): Feed the richer morphological tags from crate en into the parser features instead of a coarse tagset, and measure the LAS change.
  • I5 (tokens): Add explicit rule-based handling for numeric expressions, compounds and names in the tagger and lemmatizer so out-of-vocabulary words degrade parsing less.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GroningenGroningen · Netherlands49 works · 4,843 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Barbara Plank. 2010. Improved Statistical Measures to Assess Natural Language Parser Performance across Domains. Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{plank2010improved,
  title = {{Improved Statistical Measures to Assess Natural Language Parser Performance across Domains}},
  author = {Barbara Plank},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  doi = {10.63317/4z465k4pe3rn},
  url = {https://doi.org/10.63317/4z465k4pe3rn}
}

References

13 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Conference on Empirical Methods in Natural Language Processing, Sydney, Australia.
  2. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Association for Computational Linguistics, Prague, Czech Republic.
  3. Sabine Buchholz and Erwin Marsi. 2006. Conll-x shared task on multilingual dependency parsing. In In Proc. of CoNLL, pages 149–164.
  4. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Conference of the Association for Computational Linguistics (ACL), Prague, Czech Republic.
  5. Mark Dredze, John Blitzer, Pratha Pratim Talukdar, Kuzman Ganchev, Joao Graca, and Fernando Pereira. 2007. Frustratingly hard domain adaptation for parsing. In Proceedings of the CoNLL Shared Task Session - Conference on Natural Language Learning, Prague, Czech Alpino cor= 0.1560285 cor= −0.01164684 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● LAS LAS ● ● ● ● ● ● ● ● ● ● ● 0.000 0.010 0.020 l aUWR MST cor= −0.3914173 cor= 0.1925337 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● LAS LAS ● ● 0.15 0.25 l sUWR Malt cor= −0.2855470 cor= −0.02882270 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● LAS LAS ● ● 0.15 0.25 l sUWR Figure 2: Results: Parser performance on a per-article basis against statistical measure on the text (93 Wikipedia articles - 3 sports articles removed). Republic. find
  6. Daniel Gildea. 2001. Corpus variation and parser performance. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  7. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152–159, New York City. Association for Computational Linguistics.
  8. Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajič. 2005. Non-projective dependency parsing using spanning tree algorithms. In In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 523–530.
  9. Joakim Nivre and Ryan McDonald. 2008. Integrating graph-based and transition-based dependency parsers. In Proceedings of ACL-08: HLT, pages 950–958, Columbus, Ohio, June. Association for Computational Linguistics. cor= −0.3310151 cor= 0.0915257 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● LAS LAS ● ● ● ● ● ● 0.10 0.20 0.30 UPTR ppl_cdb cor= 0.007922983 cor= −0.6759002 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● LAS LAS ● ● ● ● ● ● 0.10 0.20 0.30 UPTR ppl_cdb cor= 0.06782834 cor= −0.5725082 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● LAS LAS ● ● ● ● 0.10 0.20 0.30 UPTR ppl_cdb Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülsen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007. Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13:95–135.
  10. Nelleke Oostdijk. 2000. The Spoken Dutch Corpus: Overview and first evaluation. In Proceedings of Second International Conference on Language Resources and Evaluation (LREC), pages 887–894. find
  11. Sujith Ravi, Kevin Knight, and Radu Soricut. 2008. Automatic prediction of parser accuracy. In EMNLP ’08: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 887–896, Morristown, NJ, USA. Association for Computational Linguistics. find
  12. Gertjan van Noord. 2006. At Last Parsing Is Now Operational. In TALN 2006 Verbum Ex Machina, Actes De La 13e Conference sur Le Traitement Automatique des Langues naturelles, pages 20–42, Leuven. find
  13. Yi Zhang and Rui Wang. 2009. Correlating natural language parser performance with statistical measures of the text. In In Proceedings of KI 2009, Paderborn, Germany. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/plank-2010-improved-statistical-measures-assess-natural/