Elamur Research⌕
☰
home / papers / 2017 / Exploring Different Dimensions of Attention for Uncertainty Detection

Exploring Different Dimensions of Attention for Uncertainty Detection

Paper · 2017 · Heike Adel and Hinrich Schütze

Summary

The paper addresses uncertainty detection (UD): deciding whether a sentence states something as fact or hedges it through speculation, opinion, vagueness or ambiguity. This matters for knowledge base population, where hedged claims should not become facts. The authors use the two CoNLL-2010 hedge cue corpora, one from Wikipedia ("wiki") and one biomedical ("bio"). Their stated first is applying CNNs and bidirectional GRU RNNs to this task. They then explore attention along three axes:

  • Weighted vs. unweighted selection: attention vs. pooling.
  • Sequence-agnostic vs. sequence-preserving: averaging vs. keeping word order.
  • Internal vs. external: where the weights come from.

External attention scores each word against a lexicon of cue phrases taken from the training set. The best CNN with external attention beats the previous wiki state of the art by over 3.5 F1. On bio it roughly matches a strong feature-rich system without handcrafted linguistic features. A small sentiment experiment suggests the ideas transfer to other tasks. The code is public.

Key points

  • Data: the CoNLL-2010 hedge cue detection dataset, with two medium-sized corpora (wiki and bio) and binary certain/uncertain labels.
  • Models:
    • The CNN has one convolutional layer, 3-max pooling, a fully connected layer and a logistic output.
    • The RNN is a bidirectional GRU with gradient clipping.
  • External attention scoring compares each word with every cue vector through a small tanh hidden layer and sums the results. Each cue vector is the averaged word embedding of a training cue phrase.
  • Three ways to connect attention downstream:
    • plain weighted average;
    • "k-max sequence" (per-pos), which keeps the top positions in their original order;
    • attention-weighted k-max pooling (per-dim).
  • Results:
    • The CNN beat the RNN overall, especially on long sentences.
    • A reimplementation of Meng et al. (2015) reached only 51.51 on wiki and 66.57 on bio, which the authors attribute to its need for much more training data.
    • An outlook test used SST-2 sentiment.

Interesting ideas

  • I1. Separating the "focus" (the layer being reweighted) from the "source" (the information used to compute weights) makes it easy to bring in outside knowledge that the network never sees directly.
  • I2. Representing a multiword cue as the average of its word embeddings lets the system recognize unseen cues that resemble known ones.
  • I3. Order-preserving selection keeps the chosen elements in sentence order instead of averaging them, so constructions like "not uncertain" vs. "uncertain that ... not" stay distinguishable.
  • I4. Weighted selection lets the model decide how many cues matter, whereas k-max pooling caps the count at k and can miss sentences with many hedges.
  • I5. The authors break performance down by sentence length to explain why one architecture wins.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): hedge and modality detection in prag could weight candidate tokens by their similarity to a curated cue lexicon ("may", "rumored", "suggest"), keeping the reason for each judgment inspectable.
  • I2 (world): when deciding whether a statement enters the world of the text as a fact or as a belief/possibility, soft matching against cue-phrase vectors could catch paraphrased hedges that are missing from the lexicon.
  • I3 (tree): this is a reminder that negation and hedge scope must be read from tree position, so negation and hedge attachment to their heads should be checked explicitly in parser evaluation.
  • I4 (learn): explicit rule-based detectors should collect all matching cues rather than the first few, and evaluation should include multi-cue sentences as a stress case.
  • I5 (learn): when comparing witness parsers or teachers, F1 bucketed by sentence length can reveal where each one degrades and guide which witness to trust.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Heike Adel and Hinrich Schütze. 2017. Exploring Different Dimensions of Attention for Uncertainty Detection. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{adel2017exploring,
  title = {{Exploring Different Dimensions of Attention for Uncertainty Detection}},
  author = {Heike Adel and Hinrich Schütze},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers},
  year = {2017},
  doi = {10.18653/v1/e17-1003},
  url = {https://doi.org/10.18653/v1/e17-1003}
}

References

50 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 626–634, Beijing, China, July. Association for Computational Linguistics. ↗
  2. Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code. In Proceedings of The 33rd International Conference on Machine Learning (ICML-16), ICML ’16, pages 2091– 2100, New York City, NY, USA, June. ACM. ↗
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations (ICLR), San Diego, California, USA, May. ↗
  4. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159. find
  5. Richárd Farkas, Veronika Vincze, György Móra, János Csirik, and György Szarvas. 2010. The conll-2010 shared task: Learning to detect hedges and their scope in natural language text. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 1–12, Uppsala, Sweden, July. Association for Computational Linguistics.
  6. Kathryn Baker, Michael Bloodgood, Bonnie J Dorr, Chris Callison-Burch, Nathaniel W Filardo, Christine Piatko, Lori Levin, and Scott Miller. 2012. Use of modality and negation in semantically-informed syntactic MT. Computational Linguistics, 38:411– 438. find
  7. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2000. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  8. Viola Ganter and Michael Strube. 2009. Finding hedges by chasing weasels: Hedge detection using wikipedia tags and shallow linguistic features. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 173–176, Suntec, Singapore, August. Association for Computational Linguistics.
  9. Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. 2015. Abc-cnn: An attention based convolutional neural network for visual question answering. Technical Report. arXiv:1511.05960. ↗
  10. Maria Georgescul. 2010. A hedgehop over a max-margin framework using hedge cues. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 26–31, Uppsala, Sweden, July. Association for Computational Linguistics.
  11. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724– 1734, Doha, Qatar, October. Association for Computational Linguistics. ↗
  12. Xiaodong He and David Golub. 2016. Character-level question answering with attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1598–1607, Austin, Texas, November. Association for Computational Linguistics. ↗
  13. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. ↗
  14. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS’15 Proceedings of the 28th International Conference on Neural Information Processing Systems, pages 1693–1701, Montreal, Canada. MIT Press. ↗
  15. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537.
  16. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 655–665, Baltimore, Maryland, June. Association for Computational Linguistics. ↗
  17. Marie-Catherine De Marneffe, Christopher D Manning, and Christopher Potts. 2012. Did it happen? the pragmatic complexity of veridicality assessment. Computational linguistics, 38:335–367. ↗
  18. Noa P. Cruz Dı́az, Maite Taboada, and Ruslan Mitkov. 2016. A machine-learning approach to negation and speculation detection for sentiment analysis. Journal of the Association for Information Science and Technology (JASIST), 67(9):2118–2136. ↗
  19. Lauri Karttunen and Annie Zaenen. 2005. Veridicity. In Graham Katz, James Pustejovsky, and Frank Schilder, editors, Annotating, Extracting and Reasoning about Time and Events, number 05151 in Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal, September. Association for Computational Linguistics. Dagstuhl Seminar Proceedings, Dagstuhl, Germany. Internationales Begegnungs- und Forschungszentrum für Informatik (IBFI), Schloss Dagstuhl, Germany.
  20. Lauri Karttunen. 1973. Presuppositions of compound sentences. Linguistic Inquiry, 4:169–196. find
  21. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746– 1751, Doha, Qatar, October. Association for Computational Linguistics.
  22. Paul Kiparsky and Carol Kiparsky. 1970. Progress in Linguistics, pages 143–173. find
  23. Roser Saurı́ and James Pustejovsky. 2009. Factbank: A corpus annotated with event factuality. Language Resources and Evaluation, 43:227–268. find
  24. Roser Saurı́ and James Pustejovsky. 2012. Are you sure that this happened? Assessing the factuality degree of events in text. Computational Linguistics, 38:261–299. Fact.
  25. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA, October. Association for Computational Linguistics.
  26. Xiujun Li, Wei Gao, and Jude W Shavlik. 2014. Detecting semantic uncertainty by learning hedge cues in sentences using an hmm. In SIGIR 2014 Workshop on Semantic Matching in Information Retrieval, Queensland, Australia, July. find
  27. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland, June. Association for Computational Linguistics. ↗
  28. Marijn F Stollenga, Jonathan Masci, Faustino Gomez, and Jürgen Schmidhuber. 2014. Deep networks with internal selective attention through feedback connections. In NIPS’14 Proceedings of the 27th International Conference on Neural Information Processing Systems, pages 3545–3553, Montreal, Canada, December. ↗
  29. Fandong Meng, Zhengdong Lu, Mingxuan Wang, Hang Li, Wenbin Jiang, and Qun Liu. 2015. Encoding source language with convolutional neural network for machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 20–30, Beijing, China, July. Association for Computational Linguistics. ↗
  30. György Szarvas, Veronika Vincze, Richárd Farkas, György Móra, and Iryna Gurevych. 2012. Cross-genre and cross-domain detection of semantic uncertainty. Computational Linguistics, 38:335–367. ↗
  31. Buzhou Tang, Xiaolong Wang, Xuan Wang, Bo Yuan, and Shixi Fan. 2010. A cascade method for detecting hedges and their scope in natural language text. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 13–17, Uppsala, Sweden, July. Association for Computational Linguistics.
  32. Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, pages 1045–1048, Makuhari, Chiba, Japan, September. find
  33. Erik Velldal, Lilja Øvrelid, Jonathon Read, and Stephan Oepen. 2012. Speculation and negation: Rules, rankers, and the role of syntax. Computational linguistics, 38:369–410. ↗
  34. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of Workshop at 1st International Conference on Learning Representations (ICLR), Scottsdale, Arizona, USA, May. ↗
  35. Veronika Vincze, György Szarvas, Richárd Farkas, György Móra, and János Csirik. 2008. The bioscope corpus: biomedical texts annotated for uncertainty, negation and their scopes. BMC bioinformatics, 9:1–9. ↗
  36. Baolin Peng, Zhengdong Lu, Hang Li, and Kam-Fai Wong. 2015. Towards neural network-based reasoning. In arXiv preprint arXiv:1508.05508. ↗
  37. Veronika Vincze. 2014a. Uncertainty detection in hungarian texts. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1844–1853, Dublin, Ireland, August. Dublin City University and Association for Computational Linguistics. find
  38. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, May. ↗
  39. Huiwei Zhou, Huan Yang, Long Chen, Zhenwei Liu, Jianjun Ma, and Degen Huang. 2015. Combining feature-based and instance-based transfer learning approaches for cross-domain hedge detection with multiple sources. In Social Media Processing, volume 568, pages 225–232. Springer. find
  40. Veronika Vincze. 2014b. Uncertainty Detection in Natural Language Texts. Ph.D. thesis, University of Szeged. ↗
  41. Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. 2016. Relation classification via multi-level attention cnns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1298– 1307, Berlin, Germany, August. Association for Computational Linguistics.
  42. Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. 2016. Text classification improved by integrating bidirectional lstm with two-dimensional max pooling. In Proceedings of COL-ING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3485–3495, Osaka, Japan, December. ↗
  43. The COLING 2016 Organizing Committee. find
  44. Zhongyu Wei, Junwen Chen, Wei Gao, Binyang Li, Lanjun Zhou, Yulan He, and Kam-Fai Wong. 2013. An empirical study on uncertainty identification in social media context. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 58–62, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  45. Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. 2015. The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 842– 850, Boston, MA, USA, June. find
  46. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480–1489, San Diego, California, June. Association for Computational Linguistics.
  47. Wenpeng Yin and Hinrich Schütze. 2015. Multichannel variable-size convolution for sentence classification. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, pages 204–214, Beijing, China, July. Association for Computational Linguistics. find
  48. Wenpeng Yin, Hinrich Schtze, Bing Xiang, and Bowen Zhou. 2016. Abcnn: Attention-based convolutional neural network for modeling sentence pairs. Transactions of the Association for Computational Linguistics, 4:259–272. ↗
  49. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335–2344, Dublin, Ireland, August. Dublin City University and Association for Computational Linguistics.
  50. Dongxu Zhang and Dong Wang. 2015. Relation classification via recurrent neural network. Technical Report. arXiv:1508.01006. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/adel-2017-exploring-different-dimensions-attention-uncertainty/