📄 A Machine Learning Approach to Automatic Functor Assignment in the Prague Dependency Treebank

Summary

Converting analytical tree structures (ATS) into tectogrammatical tree structures (TGTS) in the Prague Dependency Treebank (PDT) requires a great deal of manual work by expert linguists. The paper describes a new version of the AFAS system, which automates part of this work: assigning functors to autosemantic words. Functors are about 60 labels for arguments and adjuncts (Actor, Patient, Addressee, Manner, etc.) that specify the relation between a dependent node and its governing node. The task is simplified to edge classification: the label is predicted only from the attributes of the node and its nearest autosemantic ancestor. The features are lemmas, morphological categories, analytical functions, the number of child nodes, and concepts from the EuroWordNet Top Ontology. The data comprise 27 463 nodes from 1536 manually annotated sentences. The classifier is a C5.0 decision tree, which is automatically converted into a Perl module for the TrEd annotation editor. The system errs in roughly one case out of five. This is less than twice as often as two annotators disagree with each other. The annotators estimate that without AFAS they would spend at least twice as much time on functors. The work shows that even near-semantic annotation can be substantially accelerated by a system that relies solely on machine learning and is easy to retrain.

Key points

• Data: 43 PDT files (texts from the Czech National Corpus) that, at the time of the experiments, had already been manually annotated at the tectogrammatical level. Together they contain 1536 sentences and 28 216 autosemantic and coordination nodes. The authors removed nodes with an unambiguous functor and nodes that annotators had added without a surface counterpart (for example, an elided Actor). This left 27 463 examples for training and testing.

• Features: for the governing and the dependent node, the following are used:

  • the lemmas, as well as the lemma of the preposition or subordinating conjunction between them;
  • the part of speech and its subtype, voice, and case;
  • the analytical function;
  • the number of child nodes in the TGTS.

  In addition, 63 binary attributes from the EuroWordNet Top Ontology (concepts such as Place, Time, Human) were added. They were mapped onto Czech lemmas via the Inter Lingual Index.

• Method: C5.0 is the successor of C4.5 (Quinlan, 1993). It builds the tree top-down (TDIDT) and prunes statistically unreliable branches. The parameters were left at their defaults. Only trees were evaluated, because building rulesets takes considerably more time. Earlier versions of AFAS relied on hand-written rules and dictionaries, later supplemented with C4.5 trees. The new version is learned entirely from data.

• Results: about 20 % of the automatically assigned functors need to be corrected. The disagreement with humans is less than twice the inter-annotator disagreement, which the authors regard as a natural bound on the error rate. The error rate decreases with a larger training set. On files annotated with the previous version of AFAS, annotators changed ≈23 % of the functors, which almost exactly matches the prediction.

• Deployment and feedback: in TrEd, a single keyboard command assigns functors throughout the whole tree. Color shows the confidence of the leaf: red < 60 %, blue 60–90 %, black > 90 %, and green marks labels already corrected by hand. Six annotators who have been annotating for at least a year and have worked with AFAS for more than half a year were surveyed:

  • all of them chose maximum coverage (recall) over higher precision;
  • they apparently do not use the colors, although on new data at least three quarters of the errors fall on red labels;
  • one of their suggestions led to the addition of topological features.