Elamur Research⌕
☰
home / papers / 2016 / Incremental Fine-grained Information Status Classification Using Attention-based LSTMs

Incremental Fine-grained Information Status Classification Using Attention-based LSTMs

Paper · 2016 · Yufang Hou

Summary

The paper tackles fine-grained information status (IS) classification. Each noun phrase mention in a written text is labelled old, new, or one of six mediated subtypes: worldKnowledge, syntactic, bridging, comparative, aggregate and function. Earlier systems on ISNotes depended on many hand-built semantic features drawn from knowledge bases. Hou asks how much of that knowledge pretrained word embeddings can supply instead.

The model processes mentions one at a time, in document order, the way a reader would. Two LSTMs encode the sequence of mentions: one uses summed word embeddings, the other one-hot vectors. An attention layer then weighs a window of earlier mentions before a softmax picks one of the eight classes. Labels predicted for earlier mentions are fed forward during decoding.

With embeddings alone, the model already handles several categories reasonably. With a few simple extra features added, it comes close to the feature-heavy state of the art of Hou et al. (2013). This suggests that learned representations can replace much costly knowledge engineering in discourse processing.

Key points

  • Data: ISNotes covers 50 Wall Street Journal texts from OntoNotes, with 1,726 sentences and 10,980 annotated mentions.
  • Class balance: new is 36.7%, old is 29.5% and mediated overall is 33.8%. Within mediated, function is the rarest at 0.6%, and comparative and bridging are also scarce.
  • Architecture: two separate LSTMs (embedding-sum and one-hot mention encodings) worked better than one LSTM over concatenated inputs. Attention covers preceding-mention windows of 10, 20 or 50.
  • Ablation: removing predicted IS labels of earlier mentions from the best model (LSTM+PAR+FEAT) lowers accuracy from 78.6 to 77.7. The losses fall mostly on mediated/syntactic, mediated/aggregate and new.
  • Attention behaviour: heat maps show attention landing on plausible antecedents, for example "Saudi Arabia" for "the kingdom". It also lands on syntactic children for possessive and coordinated mentions. For bridging and comparative, no clear pattern appears, probably because training data is sparse.

Interesting ideas

  • I1. Inner-first ordering: mentions are sorted by end position, and ties are broken by start position descending, so nested mentions are handled before the phrases that contain them. This lets a parent's label depend on its children's labels.
  • I2. Incremental decoding with earlier predictions: the label already assigned to an earlier mention becomes an input when classifying later ones, mimicking left-to-right reading and modelling dependencies between labels.
  • I3. Dummy start mention: a placeholder mention at the start of each document gives attention a "nothing relevant" target. As a result, the first mention, or one without an antecedent, still has a well-formed context.
  • I4. Reading attention as evidence: attention weights were inspected to see whether the model points to real antecedents or syntactic children. Spread-out weights signalled that no antecedent was present in the window.
  • I5. Status taxonomy with mediated subtypes: separating bridging, comparative, function and aggregate mediation gives a finer account of why a referent is accessible.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): When building frames and coreference over nested NPs, process inner spans first, so a possessive or coordination node can inherit status from its parts.
  • I2 (world): Entity status in the text world (introduced, known, inferable) could be updated step by step, with each new mention conditioned on statuses already committed.
  • I3 (frames): A null-antecedent slot in coreference and bridging decisions makes "new entity" an explicit, explainable choice rather than a fallback.
  • I4 (learn): In evaluation, check whether the mention a component relies on is the true antecedent, and treat diffuse support as a negative-control signal for unreliable cases.
  • I5 (world): Tag each entity in the text world with how it became accessible (world knowledge, part of something mentioned, a value of a function such as a share price), so answers can explain referent resolution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heidelberg UniversityHeidelberg · Germany27 works · 40,944 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Yufang Hou. 2016. Incremental Fine-grained Information Status Classification Using Attention-based LSTMs. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hou2016incremental,
  title = {{Incremental Fine-grained Information Status Classification Using Attention-based LSTMs}},
  author = {Yufang Hou},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  url = {https://aclanthology.org/C16-1177/}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference for Learning Representations (ICLR 2015), San Diego, 2015. ↗
  2. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of the 17th International Conference on Computational Linguistics and 36th Annual Meeting of the Association for Computational Linguistics, Montréal, Québec, Canada, 10–14 August 1998, pages 86–90.
  3. Stefan Baumann and Arndt Riester. 2013. Coreference, lexical givenness and prosody in German. Lingua. Accepted. ↗
  4. Betty J. Birner and Gregory Ward. 1998. Information Status and Noncanonical Word Order in English. John Benjamins, Amsterdam, The Netherlands. ↗
  5. Aoife Cahill and Arndt Riester. 2009. Incorporating information status into generation ranking. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing, Singapore, 2–7 August 2009, pages 817–825.
  6. Aoife Cahill and Arndt Riester. 2012. Automatically acquiring fine-grained information status distinctions in German. In Proceedings of the SIGdial 2012 Conference: The 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Seoul, Korea, 5–6 July 2012, pages 232–236. find
  7. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, U.K., 27–29 July 2011. find
  8. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, Mass. find
  9. Katja Filippova, Enrique Alfonseca, Carlos Colmenares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with LSTMs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, 17–21 September 2015, pages 360–368.
  10. Jeanette K. Gundel, Nancy Hedberg, and Ron Zacharski. 1993. Cognitive status and the form of referring expressions in discourse. Language, 69:274–307. ↗
  11. M. A. K. Halliday. 1967. Notes on transitivity and theme in English, Part 2. Journal of Linguistics, 3:199–244. find
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  13. Yufang Hou, Katja Markert, and Michael Strube. 2013. Cascading collective classification for bridging anaphora recognition using a rich linguistic feature set. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Wash., 18–21 October 2013, pages 814–820.
  14. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations (ICLR 2015), San Diego, 2015. ↗
  15. Ivana Kruijff-Korbayová and Mark Steedman. 2003. Discourse and information structure. Journal of Logic, Language and Information. Special Issue on Discource and Information Structure, 12(3):149–259. ↗
  16. Knud Lambrecht. 1994. Information Structure and Sentence Form. Cambridge, U.K.: Cambridge University Press. ↗
  17. Katja Markert, Yufang Hou, and Michael Strube. 2012. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju Island, Korea, 8–14 July 2012, pages 795–804.
  18. Quinn McNemar. 1947. Note on the sampling errors of the difference between correlated proportions or percentages. Phychometrika, 12(2):153–157. find
  19. Malvina Nissim, Shipara Dingare, Jean Carletta, and Mark Steedman. 2004. An annotation scheme for information status in dialogue. In Proceedings of the 4th International Conference on Language Resources and Evaluation, Lisbon, Portugal, 26–28 May 2004, pages 1023–1026. ↗
  20. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25–29 October 2014, pages 1532–1543.
  21. Ellen F. Prince. 1981. Towards a taxonomy of given-new information. In P. Cole, editor, Radical Pragmatics, pages 223–255. Academic Press, New York, N.Y. find
  22. Ellen F. Prince. 1992. The ZPG letter: Subjects, definiteness, and information-status. In W.C. Mann and S.A. Thompson, editors, Discourse Description. Diverse Linguistic Analyses of a Fund-Raising Text, pages 295–325. John Benjamins, Amsterdam. ↗
  23. Altaf Rahman and Vincent Ng. 2011. Learning the information status of noun phrases in spoken dialogues. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, U.K., 27–29 July 2011, pages 1069–1080.
  24. Altaf Rahman and Vincent Ng. 2012. Learning the fine-grained information status of discourse entities. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, 23–27 April 2012, pages 798–807.
  25. Arndt Riester, David Lorenz, and Nina Seemann. 2010. A recursive annotation scheme for referential information status. In Proceedings of the 7th International Conference on Language Resources and Evaluation, La Valetta, Malta, 17–23 May 2010, pages 717–722. ↗
  26. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, and Phi Blunsom. 2016. Reasoning about entailment with neural attention. In Proceedings of the 4th International Conference on Learning representations (ICLR 2016), Caribe Hilton, San Juan, Puerto Rico, 2–4 May 2016. ↗
  27. Philip J. Stone, Dexter C. Dunphy, Marshall S. Smith, Daniel M. Ogilvie, and Cambridge Computer Associates. 1966. General Inquirer: A Computer Approach to Content Analysis. MIT Press, Cambridge, Mass. find
  28. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 (NIPS 2014), pages 3104–3112. Curran Associates, Inc. ↗
  29. Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems 28 (NIPS 2015), pages 2773–2781. Curran Associates, Inc. ↗
  30. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2011. OntoNotes release 4.0. LDC2011T03, Philadelphia, Penn.: Linguistic Data Consortium. ↗
  31. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32th International Conference on Machine Learning, Lille, France, 6–11 July 2015, pages 2048–2057. ↗
  32. Mo Yu and Mark Dredze. 2015. Learning composition models for phrase embeddings. TACL, 3:227–242. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hou-2016-incremental-fine-grained-information-status/