📄 UPOS-DEPREL Mismatches: Detecting Annotation Errors and Improving UD Guidelines based on Linguistic Knowledge

Summary

Studies of UD guidelines usually examine one annotation layer at a time. This paper looks at how two layers interact: the universal part-of-speech tag (UPOS) and the dependency relation (DEPREL) on the same token. The authors cross-tabulate UPOS against DEPREL in the 20 PUD treebanks of UD v2.10. They find that the resulting matrices are sparse, that the frequencies of attested pairs follow a long-tail distribution, and that most parts of speech have one dominant relation. They explain this pattern with Croft's model, which crosses semantic classes with propositional acts.

From the dominant pairings and the attested minor pairings that reflect category shift (transcategorization), they define a Possible Combination Zone. Any pair outside the zone is flagged as one of three things:

• an annotation error,
• an inconsistency between annotators or treebanks,
• a construction that UD describes poorly.

This yields an error detector grounded in linguistic theory. Two case studies then propose guideline changes, one for multiword expressions and one for modifier relations. The work matters because it gives treebank maintainers a cheap, interpretable consistency check and a principled basis for revising tags.

Key points

• Data. PUD (UD v2.10) covers 20 languages with 1,000 parallel news and Wikipedia sentences each. Treebank sizes range from 15,813 tokens (Finnish) to 28,788 (Japanese). DEPREL inventories range from 25 labels (Japanese) to 58 (Polish).
• Utilization rate (UR). UR is the share of attested pairs among all theoretically possible pairs. It is low everywhere, from 17.9% (Indonesian) to 31.6% (Icelandic). English uses 209 of 799 possible pairs (26.2%).
• Two-way and one-way matches.
  • Two-way matches include DET–det, ADJ–amod, ADV–advmod, NUM–nummod, CCONJ–cc, SCONJ–mark, ADP–case and PUNCT–punct.
  • Nominals align with argument relations once nsubj, obj, iobj and obl are merged.
  • One-way matches include VERB with acl, advcl, xcomp and ccomp, PROPN with flat, and PRON with expl.
• Problem tags.
  • PART has no consistent matching relation across languages, which suggests the category is a catch-all.
  • AUX splits between cop and aux, which together account for about 95% of its uses.
  • Nouns appear surprisingly often as modifiers.
• Proposed guideline changes.
  • For fixed and flat MWEs, give the first token the UPOS of the whole expression (e.g. in addition → ADV), mask internal UPOS, and concatenate the lemma, following Yih's CUD.
  • For modifiers, either merge them as SUD does with mod, or remap them to NP layers (deictics, numeratives, epithets, classifiers).

Interesting ideas

• I1. Use a UPOS×DEPREL contingency table with a whitelist of attested pairs as a rule-based checker, so that unusual pairs such as ADP & obj surface as likely errors without any trained model.
• I2. Separate two-way matches from one-way matches, because relations outnumber tags and a pair can be dominant in only one direction.
• I3. Read a tag that never settles on a dominant partner, like PART, as evidence that the category itself is incoherent rather than that annotators are careless.
• I4. Treat a fixed MWE as one unit: the first token carries the category of the whole expression and the full lemma, while the inner tokens are masked.
• I5. Encode finer semantic distinctions through deliberate, documented tag mismatches, such as associative adjectives as ADJ & nmod or fuzzy quantifiers as ADJ & nummod.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): Run the whitelist after the perceptron parser and gparse to flag implausible tag–relation pairs before trees reach frames.
• I2 (learn): When witness parsers build silver data, give lower confidence to tokens whose pair falls outside the learned zone, and treat those tokens as candidates for self-induced repair rules.
• I3 (tokens): Audit the tagger's own tag inventory this way to find catch-all tags worth splitting, such as AUX into copula and auxiliary.
• I4 (frames): Expressions like in addition or in theory would reach frame building as single adverbials, so their inner nouns would not be mistaken for referents.
• I5 (world): Mark political-type adjectives as classifying rather than describing, and treat many/much as quantity values rather than entity properties.