Elamur Research⌕
☰
home / papers / 2019 / Fine-grained Information Status Classification Using Discourse Context-Aware Self-Attention

Fine-grained Information Status Classification Using Discourse Context-Aware Self-Attention

Paper · 2019 · Yufang Hou

Summary

The paper addresses fine-grained information status (IS) classification. Each noun-phrase mention is labelled old, new, or one of six mediated subtypes. One of those subtypes is bridging, and detecting it is the first step of bridging resolution. Earlier strong systems on ISNotes relied on dozens of hand-engineered lexico-semantic features. Hou replaces them with a transformer initialized from BERT BASE and fine-tuned on a constructed "pseudo sentence" for every mention. That input holds two flag tokens recording string or head overlap with earlier mentions, the sentence containing the mention, a delimiter, the mention itself, and a special classification token whose hidden state yields the label. On ISNotes, evaluated with 10-fold cross-validation over documents, the full model reaches 83.0% overall accuracy. That beats the collective and cascaded collective baselines of Hou et al. (2013a) by 4.1 and 4.4 points. A variant using only the sentence context improves bridging recognition by 3.9 F1. The result shows that context-aware pretrained encoders can match or surpass carefully designed linguistic feature sets for discourse-level mention classification.

Key points

  • Data: ISNotes (Markert et al., 2012) has 10,980 annotated mentions across 50 news texts. Mentions split into roughly 29.5% old, 33.8% mediated and 36.7% new, and the rarest subclass, m/function, has only 65 examples.
  • Model setup: BERT BASE with 12 blocks, 768 hidden units and 12 heads, fine-tuned for 3 epochs at learning rate 5e-5, with pseudo sentences capped at 128 tokens.
  • Ablation ladder:
    • The mention-only model already rivals the 34-feature baselines.
    • Adding the local sentence mostly helps m/bridging and new.
    • Adding the overlap flags mostly helps old and m/worldKnowledge.
  • Widening the local context with the 1 or 2 preceding sentences gave no meaningful gain.
  • The full model wins on most classes, but it loses on m/function, which the author attributes to scarce training data. Gains are significant at p < 0.01 under a randomization test.

Interesting ideas

  • I1. Pack heterogeneous evidence into a single encoder input: symbolic flags, sentence context, a separator, the target span and a prediction slot. The attention layers can then weigh mention-internal cues against context cues on their own.
  • I2. Compress the whole preceding discourse into two cheap binary signals (same string seen before, same head seen before) instead of feeding long history. This gave most of the gain for old mentions at almost no cost.
  • I3. Mention-internal form alone carries much of the label: possessive modifiers suggest syntactic anchoring, while premodifiers like "another" or "further" suggest comparison with an earlier entity.
  • I4. Build the model in nested increments (mention only, then local sentence, then history flags) and report per-class effects. This reveals which information each category actually depends on, and it showed that extra preceding sentences were not worth adding.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): A coreference or mention-status classifier could take the same layout, so the tree-derived span and the explicit flags sit side by side and attention weights remain inspectable per token.
  • I2 (world): The entity registry can expose exactly such explainable "seen-before" flags for each new mention, deciding old vs. new before any heavier reasoning.
  • I3 (world): When building the world of a text, these surface cues on a UD subtree can trigger an explicit search for an anchor or comparison entity, rather than creating a fresh entity.
  • I4 (learn): Evaluation can use such staged ablations as controls, proving that a component's gains come from the intended signal and not from leakage or noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Yufang Hou. 2019. Fine-grained Information Status Classification Using Discourse Context-Aware Self-Attention. arXiv:1908.04755.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.04755 · arXiv 1908.04755

@misc{hou2019fine,
  title = {{Fine-grained Information Status Classification Using Discourse Context-Aware Self-Attention}},
  author = {Yufang Hou},
  year = {2019},
  doi = {10.48550/arxiv.1908.04755},
  eprint = {1908.04755},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.04755}
}

References

24 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Kaiser Lukasz, and Polosukhin Illia. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NIPS 2017), pages 1–11.
  2. Betty J. Birner and Gregory Ward. 1998. Information Status and Noncanonical Word Order in English. John Benjamins, Amsterdam, The Netherlands. ↗
  3. Aoife Cahill and Arndt Riester. 2012. Automatically acquiring fine-grained information status distinctions in German. In Proceedings of the SIGdial 2012 Conference: The 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Seoul, Korea, 5–6 July 2012, pages 232–236. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  5. Jeanette K. Gundel, Nancy Hedberg, and Ron Zacharski. 1993. Cognitive status and the form of referring expressions in discourse. Language, 69:274–307. ↗
  6. M. A. K. Halliday. 1967. Notes on transitivity and theme in English, Part 2. Journal of Linguistics, 3:199–244. find
  7. Yufang Hou. 2016. Incremental fine-grained information status classification using attention-based LSTMs. In Proceedings of the 26th International Conference on Computational Linguistics, Osaka, Japan, 11–16 December 2016, pages 1880–1890.
  8. Yufang Hou. 2018a. A deterministic algorithm for bridging anaphora resolution. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October– 4 November 2018.
  9. Yufang Hou. 2018b. Enhanced word representations for bridging anaphora resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana, 1–6 June 2018, pages 1–7. ↗
  10. Yufang Hou, Katja Markert, and Michael Strube. 2013a. Cascading collective classification for bridging anaphora recognition using a rich linguistic feature set. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Wash., 18–21 October 2013, pages 814– 820. ↗
  11. Yufang Hou, Katja Markert, and Michael Strube. 2013b. Global inference for bridging anaphora resolution. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, Georgia, 9–14 June 2013, pages 907–917.
  12. Yufang Hou, Katja Markert, and Michael Strube. 2014. A rule-based system for unrestricted bridging resolution: Recognizing bridging anaphora and finding links to antecedents. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25–29 October 2014, pages 2082–2093. ↗
  13. Yufang Hou, Katja Markert, and Michael Strube. 2018. Unrestricted bridging resolution. Computational Linguistics, 44(2):237–284. ↗
  14. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, 15–20 July 2018, pages 328–339. find
  15. Ivana Kruijff-Korbayová and Mark Steedman. 2003. Discourse and information structure. Journal of Logic, Language and Information. Special Issue on Discource and Information Structure, 12(3):149– 259. ↗
  16. Knud Lambrecht. 1994. Information Structure and Sentence Form. Cambridge, U.K.: Cambridge University Press. ↗
  17. Katja Markert, Yufang Hou, and Michael Strube. 2012. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju Island, Korea, 8–14 July 2012, pages 795–804.
  18. Malvina Nissim, Shipara Dingare, Jean Carletta, and Mark Steedman. 2004. An annotation scheme for information status in dialogue. In Proceedings of the 4th International Conference on Language Resources and Evaluation, Lisbon, Portugal, 26–28 May 2004, pages 1023–1026. ↗
  19. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana, 1–6 June 2018, pages 2227–2237. ↗
  20. Ellen F. Prince. 1981. Towards a taxonomy of given-new information. In P. Cole, editor, Radical Pragmatics, pages 223–255. Academic Press, New York, N.Y. find
  21. Ellen F. Prince. 1992. The ZPG letter: Subjects, definiteness, and information-status. In W.C. Mann and S.A. Thompson, editors, Discourse Description. Diverse Linguistic Analyses of a Fund-Raising Text, pages 295–325. John Benjamins, Amsterdam. ↗
  22. Altaf Rahman and Vincent Ng. 2012. Learning the fine-grained information status of discourse entities. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, 23–27 April 2012, pages 798–807.
  23. Arndt Riester, David Lorenz, and Nina Seemann. 2010. A recursive annotation scheme for referential information status. In Proceedings of the 7th International Conference on Language Resources and Evaluation, La Valetta, Malta, 17–23 May 2010, pages 717–722. ↗
  24. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October– 4 November 2018, pages 5027–5038.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hou-2019-fine-grained-information-status-classification/