Prepositional Phrase Attachment through a Backed-off Model

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W95-0103.pdf

Summary

The paper addresses prepositional phrase attachment: deciding whether a PP in a "verb, noun phrase, PP" sequence modifies the verb or the object noun. The decision uses only four head words: verb, first noun, preposition and the noun inside the PP. Collins and Brooks treat this as a conditional estimation problem, much like n-gram language modelling. They adapt Katz-style backed-off estimation: when a full four-word tuple was never seen in training, the model falls back to sub-tuples. Two adaptations matter. Only sub-tuples that include the preposition are used, and counts from several sub-tuples at the same level are pooled. Training and evaluation use the IBM data extracted from the Wall Street Journal Treebank, the same split as [RRR94]. The method reaches 84.5% accuracy, beating the maximum entropy and transformation-based systems evaluated on the same data. This is close to the 88.2% that human experts achieve from the four head words alone. Unexpectedly, rare training events turn out to be very valuable. The approach is simple, cheap to compute, and became a strong reference baseline.

Key points

• Data: IBM quintuples (attachment label, v, n1, p, n2) from the WSJ Treebank, with 20,801 training, 3,097 test and 4,039 development examples. About 95% of test quadruples never occur in training.
• Bounds: Always choosing noun attachment gives 59.0%. Picking the most frequent attachment per preposition gives 72.2%. Human experts score 88.2% with the four head words and 93.2% with the full sentence.
• Comparison on the IBM data:
  • [RRR94]: 77.7% with words only and 81.6% with word classes.
  • [BR94] transformations: 81.9%.
  • Backed-off model: 84.1% on raw data and 84.5% with morphological processing.
• Low counts matter: Zeroing out every count below 5 lowers accuracy from 84.1% to 81.6%. The best cut-off threshold at every back-off stage was 0.
• Versus Hindle & Rooth: On a restricted subset of 1,924 test cases, their decision rule scores 82.1%, while the backed-off measure scores 86.5%.

Interesting ideas

• Reframe the task as an n-gram problem. Treating a classification decision over a few context words as smoothed conditional estimation lets well-tested language-modelling machinery carry over. The same move could apply to other small-context disambiguation tasks, such as case or agreement choices in Ukrainian.
• Constrain the back-off space with domain knowledge. When the context has no natural order, the number of possible back-off paths grows quickly. Keeping only sub-tuples that contain the most informative element (here the preposition) both shrinks the search and raises accuracy. A per-sub-tuple ablation confirmed this: all seven preposition-containing tuples ranked above all others.
• Pool counts instead of averaging estimates. Summing the numerators and denominators of several same-level sub-tuples automatically trusts better-attested evidence more than a plain average does. This is a cheap, parameter-free way to combine sparse statistics.
• Test frequency cut-offs before assuming them. The common practice of discarding rare events actually hurt here. Any sparse-data pipeline should check this empirically rather than assume rare events are noise.
• Report a bracket of reference scores. Placing results between a trivial lower bound (most frequent label per preposition) and a human ceiling under the same input restrictions makes the scores much easier to interpret. This is useful when building new benchmarks.