📄 Annotating FrameNet via Structure-Conditioned Language Generation

Summary

FrameNet's coverage is incomplete: of 13,640 lexical units (LUs), only about 62% have annotated example sentences. The authors ask whether large language models can produce new sentences that respect a given frame-semantic structure and so fill this gap automatically. They start from the sister-LU substitution method of Pancholy et al. (2021), which swaps one verb LU for another in the same frame. They then select core frame-element (FE) spans that are likely to clash with the new LU and regenerate them. Generation uses either fine-tuned T5-large or prompted GPT-4 Turbo, conditioned on no labels, on the FE type, or on both frame and FE. A SpanBERT FE classifier then removes any output whose FE labels drift. Richer semantic conditioning combined with this filtering gives more fluent sentences that humans accept more often than plain substitution. Used as training data for frame-semantic role labeling, the generated annotations help noticeably when little data is available but add almost nothing at full scale. This is a useful caution about relying on LLMs to automate linguistic annotation.

Key points

• Candidate FEs for regeneration: a span is chosen only if it belongs to a core FE type and either has no Agent/Self-mover ancestor or is a prepositional phrase. The study is limited to verb LUs, where the mismatch problem was worst.
• Filtering: the SpanBERT FE classifier reaches 95% accuracy on FrameNet 1.7. Only generations with perfect "FE fidelity" are kept.
• Intrinsic evaluation: one sentence was generated for each of 1,000 unannotated LUs, and 200 were judged by hand. After filtering, GPT-4 with FE or Frame+FE conditioning reached roughly 0.82–0.83 human acceptance. Unconditioned GPT-4 scored about 0.53 before filtering.
• Full-data frame-SRL: augmentation barely moved overall F1 (0.677 → 0.683 with T5 | FE, 0.684 with GPT-4 | Frame+FE). It slightly hurt the augmented LUs in one setting. The authors attribute this to diminishing returns and to low diversity in the generated sentences.
• Low-resource frame-SRL: adding 6.25% generated data to 25% of the full-text training set outperformed adding the same amount of human-annotated data. Learning curves were also run with the parser of Lin et al. (2021). Code is available at github.com/X-F-Cui/FrameNet-Conditional-Generation.