Elamur Research⌕
☰
home / papers / 2021 / Infusing Finetuning with Semantic Dependencies

Infusing Finetuning with Semantic Dependencies

Paper · 2021 · Zhaofeng Wu, Hao Peng and Noah A. Smith

Summary

Pretrained language models clearly encode a lot of syntax. Whether they also capture meaning, specifically predicate-argument structure, is less clear. The authors test this with a new full-graph probe for semantic dependency parsing on the DM formalism (CoNLL 2015 data). They compare it with a syntactic probe on the Penn Treebank converted to Stanford Dependencies. For RoBERTa and BERT, a linear probe over frozen representations falls much further below a finetuned ceiling parser on DM than on syntax. This suggests semantic structure stays largely implicit in these models.

Building on this, they propose SIFT (semantics-infused finetuning). Sentences are first parsed into DM graphs. A relational graph convolutional network (RGCN) then encodes these graphs alongside RoBERTa during task finetuning. A cheaper SIFT-Light variant needs no parser at inference time.

On GLUE, SIFT consistently beats both base and large RoBERTa baselines. The gains are biggest when finetuning data is scarce. The work shows that general-purpose linguistic supervision can still add value on top of pretraining.

Key points

  • Probing data and gaps: PTB SD has 39,832 training and 2,416 test sentences; DM has 33,964 and 1,410.

    • With RoBERTa-base, the probe drops from 95.2 to 81.7 LAS on syntax (−14.2% relative).
    • On DM it drops from 94.2 to 70.7 labeled F1 (−24.9% relative).
    • With RoBERTa-large the gaps are −18.5% for syntax and −28.3% for DM.

  • DM formalism: DM is derived from MRS/HPSG. It represents a sentence as a labeled directed graph with 59 relation labels. Unlike most syntactic trees, it can skip semantically empty words.

  • GLUE averages (base size): SIFT 87.0, SIFT-Light 86.7, RoBERTa 86.4. SIFT also outperforms other ways of injecting semantics:

    • GCN 86.8
    • GAT 86.5
    • Hidden 86.4
    • Scaffold 85.5

    Modeling relation labels helps, though GCN beats SIFT on CoLA.

  • Low-data MNLI: with only 392 training examples (0.1%), SIFT improves on RoBERTa by 2.5 points in-domain (58.7 → 61.2) and 3.2 points out-of-domain (60.5 → 63.7). With full data the gain is just 0.2–0.4.

  • Code: available at github.com/ZhaofengWu/SIFT. Diagnostic analyses use HANS and the GLUE diagnostic set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Zhaofeng Wu, Hao Peng and Noah A. Smith. 2021. Infusing Finetuning with Semantic Dependencies. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00363

@article{wu2021infusing,
  title = {{Infusing Finetuning with Semantic Dependencies}},
  author = {Zhaofeng Wu, Hao Peng and Noah A. Smith},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00363},
  url = {https://doi.org/10.1162/tacl_a_00363}
}

References

5 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (UCCA). In Proceedings of ACL.
  2. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of ICLR.
  3. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of ACL. DOI: https://doi.org/10.3115/980845.980860 ↗
  4. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. 2006. The second PASCAL recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. find
  5. Jasmijn Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima’an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of EMNLP. DOI: https://doi.org/10 .18653/v1/D17-1209 \#\# 8 Conclusion

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2021-infusing-finetuning-semantic-dependencies/