Elamur Research⌕
☰
home / papers / 2020 / Neural Mention Detection

Neural Mention Detection

Paper · 2020 · Juntao Yu, Bernd Bohnet and Massimo Poesio

Summary

Mention detection means finding every span that refers to an entity, including pronouns, common-noun phrases and nested spans, not only proper names. Most coreference pipelines either relied on rules, such as the Stanford deterministic system or taking all parser NPs, or hid detection inside end-to-end models that never expose their candidates. The authors build three standalone neural detectors and compare them. Each treats every span up to a maximum width as a yes/no mention decision:

  • LeeMD is the mention component of Lee et al. (2018), retrained on its own with sigmoid cross entropy.
  • BiaffineMD uses ELMO embeddings, a sentence-level bidirectional LSTM and a biaffine classifier.
  • BertMD uses BERT encodings plus a feed-forward scorer.

Every model can output mentions in a HIGH F1 mode (probability threshold) or a HIGH RECALL mode (fixed mention-per-word ratio). The models were tested on CONLL 2012, CRAC 2018 and GENIA. BiaffineMD performs best: it clearly beats the earlier neural detector, improves two coreference systems when its mentions are fed in, and reaches state-of-the-art nested NER. A reusable detector of this kind serves annotation projects as well as downstream systems.

Key points

  • Three architectures were compared: LeeMD, BiaffineMD and BertMD. The code is released at github.com/juntaoy/dali-md.
  • In HIGH RECALL mode, BiaffineMD gains up to 1.8 p.p. mention recall over a strong baseline. In HIGH F1 mode, it gains 5.3 (CONLL) and 6.2 (CRAC) p.p. mention F1 over the neural detector of Poesio et al. (2018).
  • Feeding its predicted mentions into coreference systems helps both kinds tested. The Clark and Manning (2016a) pipeline improves by up to 1.7 p.p. and the Lee et al. (2018) end-to-end system by up to 0.7 p.p.
  • On the GENIA corpus for nested NER, the model matches or exceeds specialised state-of-the-art systems, although it was not designed for that task.
  • HIGH F1 keeps spans whose probability exceeds a threshold β (e.g. 0.5). HIGH RECALL keeps a fixed number of top spans per word (ratio λ), following Lee et al. (2018).

Interesting ideas

  • I1. One trained scorer can serve two consumers through its decision rule: a probability cutoff for precise output, or a fixed top-k-per-word budget for generous candidate lists that a later stage will prune.
  • I2. Framing detection as a binary label over all spans up to a maximum width handles nested mentions naturally, without the limits of sequence tagging.
  • I3. The authors take a component out of a joint end-to-end model and retrain it alone with its own loss, which turns hidden intermediate output into an inspectable, reusable module.
  • I4. They evaluate the detector both directly and by plugging its output into two different downstream coreference systems, which shows whether upstream gains actually carry through.
  • I5. The biaffine scorer, borrowed from dependency parsing, works well for scoring start/end token pairs as spans. The same pairwise machinery serves both heads and span boundaries.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the coreference step in prag could take a wide, ratio-based candidate list, while the stricter thresholded list feeds anything that must be precise.
  • I2 (world): nested referents such as "the king's daughter" (both the king and the daughter) could be scored as separate span candidates, each traceable to its score, before becoming entities.
  • I3 (learn): when a teacher model handles mentions jointly, its mention layer could be distilled into a standalone stage so every candidate span stays explainable.
  • I4 (learn): mention or tree improvements should be checked by rerunning the downstream readers, not only by counting span accuracy.
  • I5 (tree): gparse's biaffine layer and encoder could be reused to score mention spans alongside arcs, sharing computation with the UD parse.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓Queen Mary University of LondonLondon · United Kingdom39 works · 3,307 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Coreference resolution · 550 works

Cite

Cite as Juntao Yu, Bernd Bohnet and Massimo Poesio. 2020. Neural Mention Detection. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yu2020neural,
  title = {{Neural Mention Detection}},
  author = {Juntao Yu, Bernd Bohnet and Massimo Poesio},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.1/}
}

References

35 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alex, B., Haddow, B., and Grover, C. (2007). Recognising nested named entities in biomedical text. In Proc. of BioNLP, pages 65–72. find
  2. Björkelund, A. and Kuhn, J. (2014). Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 47–57.
  3. Chamberlain, J., Poesio, M., and Kruschwitz, U. (2016). Phrase detectives corpus 1.0 crowdsourced anaphoric coreference. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may. European Language Resources Association (ELRA).
  4. Chiu, J. P. and Nichols, E. (2016). Named entity recognition with bidirectional lstm-cnns. Transactions of the Association for Computational Linguistics, 4:357–370. ↗
  5. Clark, K. and Manning, C. D. (2015). Entity-centric coreference resolution with model stacking. In Association for Computational Linguistics (ACL).
  6. Clark, K. and Manning, C. D. (2016a). Deep reinforcement learning for mention-ranking coreference models. In Empirical Methods on Natural Language Processing (EMNLP).
  7. Clark, K. and Manning, C. D. (2016b). Improving coreference resolution by learning entity-level distributed representations. In Association for Computational Linguistics (ACL).
  8. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493–2537.
  9. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics.
  10. Dozat, T. and Manning, C. (2017). Deep biaffine attention for neural dependency parsing. In Proceedings of 5th International Conference on Learning Representations (ICLR).
  11. Finkel, J. R. and Manning, C. D. (2009). Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 141–150, Singapore, August. Association for Computational Linguistics.
  12. Ju, M., Miwa, M., and Ananiadou, S. (2018). A neural layered model for nested named entity recognition. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1446–1459, New Orleans, Louisiana, June. Association for Computational Linguistics. ↗
  13. Kantor, B. and Globerson, A. (2019). Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673–677, Florence, Italy, July. Association for Computational Linguistics. ↗
  14. Kim, J.-D., Ohta, T., Tateisi, Y., and Tsujii, J. (2003). GENIA corpusa semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl 1 ) : i180 − −i182, 07. find
  15. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016). Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260–270. Association for Computational Linguistics. ↗
  16. Lee, H., Chang, A., Peirsman, Y., Chambers, N., Surdeanu, M., and Jurafsky, D. (2013). Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. ↗
  17. Lee, K., He, L., Lewis, M., and Zettlemoyer, L. (2017). End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
  18. Lee, K., He, L., and Zettlemoyer, L. S. (2018). Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Annual Conference of the North American Chapter of the Association for Computational Linguistics.
  19. Lin, H., Lu, Y., Han, X., and Sun, L. (2019). Sequence-to-nuggets: Nested entity mention detection via anchor-region networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5182–5192, Florence, Italy, July. Association for Computational Linguistics. find
  20. Lu, W. and Roth, D. (2015). Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 857–867, Lisbon, Portugal, September. Association for Computational Linguistics.
  21. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  22. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. S. (2018). Deep contextualized word representations. In Proceedings of the 2018 Annual Conference of the North American Chapter of the Association for Computational Linguistics.
  23. Poesio, M., Stuckardt, R., and Versley, Y. (2016). Anaphora Resolution: Algorithms, Resources and Applications. Springer, Berlin. find
  24. Poesio, M., Grishina, Y., Kolhatkar, V., Moosavi, N., Roesiger, I., Roussel, A., Simonjetz, F., Uma, A., Uryupina, O., Yu, J., and Zinsmeister, H. (2018). Anaphora resolution with the arrau corpus. In Proc. of the NAACL Worskhop on Computational Models of Reference, Anaphora and Coreference (CRAC), pages 11–22, New Orleans, June.
  25. Poesio, M., Chamberlain, J., Paun, S., Yu, J., Uma, A., and Kruschwitz, U. (2019). A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics. ↗
  26. Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., and Zhang, Y. (2012). CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea.
  27. Sohrab, M. G. and Miwa, M. (2018). Deep exhaustive model for nested named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2843–2849, Brussels, Belgium, October-November. Association for Computational Linguistics.
  28. Straková, J., Straka, M., and Hajic, J. (2019). Neural architectures for nested NER through linearization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5326–5331, Florence, Italy, July. Association for Computational Linguistics.
  29. Uryupina, O. and Moschitti, A. (2013). Multilingual mention detection for coreference resolution. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 100–108. find
  30. Uryupina, O., Artstein, R., Bristot, A., Cavicchio, F., Delogu, F., Rodriguez, K. J., and Poesio, M. (2019). Annotating a broad range of anaphoric phenomena, in a variety of genres: the ARRAU corpus. Journal of Natural Language Engineering. ↗
  31. Wiseman, S., Rush, A. M., Shieber, S., and Weston, J. (2015). Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1416–1426.
  32. Wiseman, S., Rush, A. M., and Shieber, S. M. (2016). Learning global features for coreference resolution. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 994–1004.
  33. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  34. Zhang, R., Nogueira dos Santos, C., Yasunaga, M., Xiang, B., and Radev, D. (2018). Neural coreference resolution with deep biaffine attention by joint mention detection and mention clustering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 102–107. Association for Computational Linguistics. ↗
  35. Zheng, C., Cai, Y., Xu, J., Leung, H.-f., and Xu, G. (2019). A boundary-aware neural model for nested named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 357–366, Hong Kong, China, November. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2020-neural-mention-detection/