Elamur Research⌕
☰
home / papers / 2023 / PairSpanBERT: An Enhanced Language Model for Bridging Resolution

PairSpanBERT: An Enhanced Language Model for Bridging Resolution

Paper · 2023 · Hideo Kobayashi, Yufang Hou and Vincent Ng

Summary

Bridging resolution means finding anaphors that relate to an earlier entity through a non-identity link, such as part-whole or attribute, and connecting them to that antecedent. Annotated data for this task is scarce. The authors take SpanBERT and continue pre-training it with its original objectives (MLM and SBO) plus a new objective, ANO. ANO trains the model to recognise the kinds of surrounding text where an unstated relation holds between a pair of mentions. The training signal comes from large amounts of noisy, automatically labelled text, built from noun pairs found in two ways: heuristically through syntactic structures, and by distant supervision from ConceptNet. The resulting model, PairSpanBERT, replaces SpanBERT inside an existing multi-task resolver. That resolver jointly handles coreference and bridging and can add precision-weighted predictions from a rule-based resolver. Evaluation covers ISNotes, BASHI and ARRAU RST, both end-to-end from raw text and with gold mentions. The new encoder gives the best published results on all three datasets in both settings. The work shows that a pre-training task tailored to the target task can pay off even when the automatic labels are noisy.

Key points

  • PairSpanBERT achieves state-of-the-art bridging results on ISNotes, BASHI and ARRAU RST, in both the end-to-end and gold-mention settings. The authors present this as the first end-to-end bridging evaluation on ARRAU RST.
  • On the dev set, combining noun pairs from syntactic structures and from ConceptNet always beat using either source alone.
  • Control experiment: SpanBERT was pre-trained further with MLM and SBO alone for the same number of epochs. Its scores barely changed, so the gains come from ANO rather than from extra training.
  • Error analysis of the best end-to-end models:
    • Anaphor recognition recall stays low, between 25.6% and 39.5%.
    • Definite NPs are missed more often than other anaphors (31% vs 45% recall on ISNotes).
    • Many false positives are new mentions wrongly labelled as bridging (43% of precision errors on ISNotes, 71% on ARRAU RST).
  • Comparison with the previous best system:
    • On ISNotes, recognition recall rises from 31.6% to 39.5%, with gains mainly on bare NPs.
    • On BASHI, recognition precision rises from 36.0% to 43.0%.
  • The checkpoint is published on Hugging Face as utd/pairspanbert.

Interesting ideas

  • I1. Design a pre-training objective that mirrors the downstream task (here, implicit links between mention pairs), then feed it large volumes of cheap, noisy labels instead of small clean ones.
  • I2. Run a matched control by giving the baseline the same amount of extra training without the new objective, so the improvement can be credited to the objective rather than to compute.
  • I3. Add a rule's measured precision on training data, scaled by a tuned weight, to the neural pair score. Rules then shift the ranking softly instead of overriding it.
  • I4. Use a knowledge graph (ConceptNet) as distant supervision for which noun pairs are likely to be implicitly related, such as an object and its parts or attributes.
  • I5. Break errors down by mention form (definite vs bare NP, new vs old mentions) and evaluate both end-to-end and with gold mentions to see where the pipeline really fails.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data agreed by witness parsers could fuel an auxiliary objective that matches the target stage, such as relations between noun pairs, rather than generic training.
  • I2 (learn): Evaluations with negative controls should include an equal-budget run like this before any new training signal is credited with a gain.
  • I3 (frames): Coreference and bridging candidates could combine learned scores with precision-weighted rule votes, keeping each link's justification traceable to a named rule.
  • I4 (world): Compiled WordNet/Wikidata relations could propose part-whole or attribute links between text entities, so that "the prices" attaches to previously mentioned goods.
  • I5 (frames): Anaphora and bridging errors in prag could be reported by NP type and by input setting, to separate mention-detection failures from linking failures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at DallasRichardson · United States61 works · 5,614 citations🔬IBM Research - IrelandDublin · Ireland11 works · 904 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksCoreference resolution · 574 works

Cite

Cite as Hideo Kobayashi, Yufang Hou and Vincent Ng. 2023. PairSpanBERT: An Enhanced Language Model for Bridging Resolution. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.383

@inproceedings{kobayashi2023pairspanbert,
  title = {{PairSpanBERT: An Enhanced Language Model for Bridging Resolution}},
  author = {Hideo Kobayashi, Yufang Hou and Vincent Ng},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.383},
  url = {https://doi.org/10.18653/v1/2023.acl-long.383}
}

References

49 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 520–534, Online. Association for Computational Linguistics. ↗
  2. Francis Bond and Ryan Foster. 2013. Linking and extending an open multilingual Wordnet. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1352–1362, Sofia, Bulgaria. Association for Computational Linguistics.
  3. Herbert H. Clark. 1975. Bridging. In Proceedings of the 1975 Workshop on Theoretical Issues in Natural Language Processing, TINLAP ’75, page 169–174, USA. Association for Computational Linguistics. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Luca Di Liello, Siddhant Garg, Luca Soldaini, and Alessandro Moschitti. 2022. Pre-training transformer models with sentence-level objectives for answer sentence selection. arXiv preprint arXiv:2205.10455. ↗
  6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. ↗
  7. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  8. Yufang Hou. 2018a. A deterministic algorithm for bridging anaphora resolution. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1938–1948, Brussels, Belgium. Association for Computational Linguistics.
  9. Yufang Hou. 2018b. Enhanced word representations for bridging anaphora resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 1–7, New Orleans, Louisiana. Association for Computational Linguistics.
  10. Yufang Hou. 2020a. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1428–1438, Online. Association for Computational Linguistics.
  11. Yufang Hou. 2020b. Fine-grained information status classification using discourse context-aware BERT. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6101–6112, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  12. Yufang Hou. 2021. End-to-end neural information status classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1377–1388, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  13. Yufang Hou, Katja Markert, and Michael Strube. 2013a. Cascading collective classification for bridging anaphora recognition using a rich linguistic feature set. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 814–820, Seattle, Washington, USA. Association for Computational Linguistics.
  14. Yufang Hou, Katja Markert, and Michael Strube. 2013b. Global inference for bridging anaphora resolution. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 907–917, Atlanta, Georgia. Association for Computational Linguistics.
  15. Yufang Hou, Katja Markert, and Michael Strube. 2014. A rule-based system for unrestricted bridging resolution: Recognizing bridging anaphora and finding links to antecedents. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2082–2093, Doha, Qatar. Association for Computational Linguistics.
  16. Yufang Hou, Katja Markert, and Michael Strube. 2018. Unrestricted bridging resolution. Computational Linguistics, 44(2):237–284. ↗
  17. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77.
  18. Sopan Khosla, Juntao Yu, Ramesh Manuvinakurike, Vincent Ng, Massimo Poesio, Michael Strube, and Carolyn Rosé. 2021. find
  19. The CODI-CRAC 2021 shared task on anaphora, bridging, and discourse deixis in dialogue. In Proceedings of the CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue, pages 1–15, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  20. Hongjin Kim, Damrin Kim, and Harksoo Kim. 2021. The pipeline model for resolution of anaphoric reference and resolution of entity reference. In Proceedings of the CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue, pages 43–47, Punta Cana, Dominican Republic. Association for Computational Linguistics. pages 1652–1659, Online. Association for Computational Linguistics.
  21. Shengjie Li, Hideo Kobayashi, and Vincent Ng. 2022. Neural anaphora resolution in dialogue revisited. In Proceedings of the CODI-CRAC 2022 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue, pages 32–47, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  22. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-bert: Enabling language representation with knowledge graph. Proceedings of the AAAI Conference on Artificial Intelligence, 34(03):2901–2908.
  23. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland. Association for Computational Linguistics. ↗
  24. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  25. Katja Markert, Yufang Hou, and Michael Strube. 2012. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 795–804, Jeju Island, Korea. Association for Computational Linguistics.
  26. Hideo Kobayashi, Yufang Hou, and Vincent Ng. 2022a. Constrained multi-task learning for bridging resolution. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 759–770, Dublin, Ireland. Association for Computational Linguistics. find
  27. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated Gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX), pages 95–100, Montréal, Canada. Association for Computational Linguistics.
  28. Hideo Kobayashi, Yufang Hou, and Vincent Ng. 2022b. End-to-end neural bridging resolution. In Proceedings of the 29th International Conference on Computational Linguistics, pages 766–778, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  29. Onkar Pandit, Pascal Denis, and Liva Ralaivola. 2020. Integrating knowledge graph embeddings to improve mention representation for bridging anaphora resolution. In Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference, pages 55–67, Barcelona, Spain (online). Association for Computational Linguistics. find
  30. Hideo Kobayashi, Shengjie Li, and Vincent Ng. 2021. Neural anaphora resolution in dialogue. In Proceedings of the CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue, pages 16–31, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  31. Hideo Kobayashi and Vincent Ng. 2020. Bridging resolution: A survey of the state of the art. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3708–3721, Barcelona, Spain (Online). International Committee on Computational Linguistics. ↗
  32. Hideo Kobayashi and Vincent Ng. 2021. Bridging resolution: Making sense of the state of the art. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Massimo Poesio and Ron Artstein. 2008. Anaphoric annotation in the ARRAU corpus. In Proceedings of the Sixth International Conference on Language Resources and Evaluation, pages 1170–1174.
  33. Massimo Poesio, Rahul Mehta, Axel Maroudas, and Janet Hitzeman. 2004. Learning to resolve bridging references. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 143–150, Barcelona, Spain.
  34. Ellen F. Prince. 1981. Toward a taxonomy of given-new information. In P. Cole, editor, Syntax and semantics: Vol. 14. Radical Pragmatics, pages 223–255. Academic Press, New York. find
  35. Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021. ERICA: Improving entity and relation understanding for pre-trained language models via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3350–3363, Online. Association for Computational Linguistics. ↗
  36. Altaf Rahman and Vincent Ng. 2011. Learning the information status of noun phrases in spoken dialogues. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1069–1080, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  37. Altaf Rahman and Vincent Ng. 2012. Learning the fine-grained information status of discourse entities. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 798–807, Avignon, France. Association for Computational Linguistics. Human Language Technologies, pages 3390–3406, Online. Association for Computational Linguistics.
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  39. Luis Von Ahn, Mihir Kedia, and Manuel Blum. 2006. Verbosity: a game for collecting common-sense facts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 75–78. find
  40. Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential Reasoning Learning for Language Representation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7170–7186, Online. Association for Computational Linguistics.
  41. Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021. Few-shot question answering by pretraining span selection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3066–3079, Online. Association for Computational Linguistics. find
  42. Juntao Yu, Sopan Khosla, Ramesh Manuvinakurike, Lori Levin, Vincent Ng, Massimo Poesio, Michael Strube, and Carolyn Rosé. 2022. find
  43. The CODI-CRAC 2022 shared task on anaphora, bridging, and discourse deixis in dialogue. In Proceedings of the CODI-CRAC 2022 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue, pages 1–14, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  44. Ina Rösiger. 2018. BASHI: A corpus of Wall Street Journal articles annotated with bridging links. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). ↗
  45. Juntao Yu and Massimo Poesio. 2020. Multitask learning-based neural bridging reference resolution. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3534–3546, Barcelona, Spain (Online). International Committee on Computational Linguistics. ↗
  46. Ina Rösiger, Arndt Riester, and Jonas Kuhn. 2018. Bridging resolution: Task definition, corpus resources and rule-based experiments. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3516–3528, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  47. A Push Singh. 2002. The public acquisition of commonsense knowledge. In Proceedings of AAAI Spring Symposium: Acquiring (and Using) Linguistic (and World) Knowledge for Information Access. AAAI Press, Palo Alto, California USA. find
  48. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An open multilingual graph of general knowledge. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 4444– 4451, San Francisco, California. AAAI Press. ↗
  49. Bo-Hsiang Tseng, Shruti Bhargava, Jiarui Lu, Joel Ruben Antony Moniz, Dhivya Piraviperumal, Lin Li, and Hong Yu. 2021. CREAD: Combined resolution of ellipses and anaphora in dialogues. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: ConceptNet Relation Types ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kobayashi-2023-pairspanbert-enhanced-language-model-bridging/