📄 Semantic Roles in Grammar Engineering

Summary

The authors investigate whether grammar engineering, in particular the construction of an LFG-based semantic parser for Polish, should rely on ready-made sets of semantic roles. Neo-Davidsonian representations, in which arguments are linked to the event via roles, simplify logical inference for tasks such as textual entailment. However, it is unclear whether such roles can be annotated reliably. To test this, the authors took 37 Polish verbs and 843 of their arguments. Seven annotators labeled these arguments with VerbNet roles and then with roles from Sowa's system, which was formulated as a decision tree for the experiment. Agreement turned out to be moderate and fell short of the 0.8 threshold that is considered indicative of high-quality annotation. The main reasons are as follows: the boundaries between roles are subjective, an argument often fits several roles or none at all, and the same role sometimes occurs more than once with a single verb. As an alternative, the authors propose a cheap deterministic method. Roles are approximated by the grammatical function, the voice of the verb, and the preposition and case of the argument. For languages with rich morphosyntax, this approach may be no less useful than manual annotation. However, this hypothesis has not yet been tested quantitatively.

Key points

• Experimental data: 37 verbs and 843 arguments in 393 sentences (one verb was considered in each sentence). The same 7 annotators labeled the arguments with VerbNet 3.2 roles (30 roles) and with Sowa roles (18 thematic roles).
• Agreement (Fleiss's κ): 0.617 for VerbNet and 0.648 for Sowa. 323 arguments were labeled unanimously with VerbNet and 253 with Sowa. For approximately 14% (VerbNet) and 18% (Sowa) of the arguments, no majority was reached.
• Typical confusions: Agent/Theme, Patient/Theme, Stimulus/Theme and Topic/Theme in VerbNet, as well as Agent/Effector (the question of the intentionality of the action) in Sowa. The object of the verb LECZYĆ ('to treat') received up to six different roles.
• Repeated role among the arguments of a single verb: 2.47% of sentences with VerbNet and 4.36% with Sowa. In the morphosyntactic approach, such sentences account for only 1.73% (6 of 347).
• Proposed replacement: pseudo-roles such as R0 (≈Agent), R1 (≈Patient/Theme) and R2 (≈Beneficiary/Recipient) are derived from grammatical functions and cases. The rules for this can be written in a few days, whereas a resource on the scale of VerbNet requires years of work. The weak point of the approach is ambiguous prepositional constructions such as Z [inst].