📄 Joint Learning Improves Semantic Role Labeling

Summary

Semantic role labeling (SRL) systems have mostly labeled each parse-tree node on its own, sometimes adding a separate label-sequence model. This ignores that a verb's arguments form a single structure with strong dependencies: roles rarely repeat, and their order depends on voice. Toutanova, Haghighi and Manning build a joint, discriminative log-linear model over whole argument frames. A local log-linear classifier, split into identification and classification, scores individual nodes. An exact dynamic program over the tree then finds the top non-overlapping labelings. A joint re-ranker, using features that describe the whole frame and label sequence, rescores these candidates, and its score is interpolated with the local model's score. On the February 2004 PropBank with gold-standard Penn Treebank parses, the joint model reduces F-Measure error by 32% on core arguments and 22% on all arguments compared with a strong local baseline. Gains are smaller but still present on Charniak parser output. The work shows that modeling global frame coherence pays off beyond what pairwise or Markov-style dependencies capture.

Key points

• Data: PropBank (Feb 2004 release) on Penn Treebank II trees. Training used sections 02–21, development used section 24, and testing used section 23. Discontinuous arguments were removed.
• Gold parses, identification plus classification: on CORE, F1 rose from 92.2 to 94.7 and Frame Accuracy from 80.7 to 88.2. On ARGM (core plus modifiers), F1 rose from 89.9 to 92.1 and Frame Accuracy from 71.8 to 79.4. This means frame-level error reductions of 38% and 26%.
• The oracle over the top N=20 local labelings reaches 99.2 (CORE) and 98.8 (ARGM) F1. N=30 gave little extra benefit and needed much more memory, so N=20 was used.
• On Charniak parses, CORE F1 went from 84.1 to 85.8 and Frame Accuracy from 66.5 to 72.7, which is a 10.7% and 18.5% error reduction. ARGM went from 81.4 to 82.9 F1. The main bottleneck was argument identification under parser errors.
• The final score combines the local log-probability, weighted by α, with the re-ranker log-probability. The best value was α = 0.5.

Interesting ideas

• I1. Tree-structured Viterbi for consistency constraints: the non-overlap rule is treated as a tiny "grammar" in which an ARG node cannot have ARG descendants. Bottom-up dynamic programming can then find the exact best consistent labeling with no hard pruning. The same trick applies to any span-labeling task over trees, such as NER over constituents.
• I2. Generate-then-rerank for non-factorizable features: a cheap local model proposes the top-N whole structures, and a joint model with global features picks among them. This makes otherwise intractable, very non-local dependencies usable in practice.
• I3. Interpolating the first-pass score into the re-ranker: the re-ranker alone degraded as N grew, because it had never learned to reject very bad candidates. Keeping the local score as a weighted term fixes this, which is a useful safeguard for any re-ranking pipeline.
• I4. Frame Accuracy as a metric: counting the share of predicates whose whole argument set is exactly right shows structural gains that per-argument F1 understates. This is valuable whenever downstream use needs complete, correct structures.
• I5. Oracle upper bound of the N-best list: measuring how good the best candidate in the top N could be justifies the choice of N and separates candidate-generation errors from re-ranking errors.