Elamur Research⌕
☰
home / papers / 2018 / A Report on the 2018 VUA Metaphor Detection Shared Task

A Report on the 2018 VUA Metaphor Detection Shared Task

Summary

The field of computational figurative language had no common benchmark, so results from different metaphor detectors were hard to compare. The authors organized the first shared task on word-level metaphor identification, which ran as part of NAACL 2018's Figurative Language Processing workshop. They used the VU Amsterdam Metaphor Corpus (VUA): BNC fragments from four genres, annotated with MIP-VU. The task had two tracks. One covered all content words (All POS), and the other covered verbs only. Submissions were scored on CodaLab by F1 for the metaphor class. The organizers also released two logistic regression baselines and the scripts needed to rebuild the data. Almost every team used a neural model, and the best All-POS system (THU NGN) reached F1 0.651. Still, the stronger feature-engineered baseline placed 4th overall. Verbs proved easiest and adverbs hardest. The gap between genres was larger than the gap between POS, with Academic scoring around .7 and Conversation around .5. The report gives the community a reproducible reference point and highlights genre robustness as an open problem.

Key points

  • Data and split: VUA has 117 fragments from the BNC across four genres (Academic, News, Conversation, Fiction). The organizers held out 23% of texts per genre for testing. Annotation used MIP-VU with κ > 0.8, and words tagged "function=mrw" counted as metaphors. Forms of be, do and have were excluded.
  • Participation: the Verbs track drew 32 submissions from 8 teams, and the All POS track drew 100 submissions from 11 teams. Each team could submit up to 12 models, and 8 teams wrote system papers.
  • Baselines: both are class-weighted logistic regression. Baseline 1 uses lemmatized unigrams (UL). Baseline 2 uses UL + WordNet + CCDB (concreteness difference). Baseline 2 scored 0.589 F1 on All POS, behind THU NGN (0.651), OCOTA (0.635) and bot.zen (0.617).
  • POS breakdown: every system did best on verbs, and nearly all did worst on adverbs. This mirrors how often each POS is metaphorical: verbs 30%, adjectives 18%, nouns 13%, adverbs 8%. The drop from best to worst POS averaged 9 points.
  • Rank consistency: system rankings changed little across POS. The rank correlation with overall All-POS performance was 0.94 for verbs, 0.98 for nouns, 0.92 for adjectives and 0.81 for adverbs. All data scripts and baseline features are on GitHub.

Interesting ideas

  • I1. Stratified test split: the test set was drawn separately within each genre. This keeps every domain represented in evaluation and makes per-genre error analysis possible, which suits any multi-domain corpus.
  • I2. Licensing workaround: instead of shipping the restricted corpus file, the organizers published a script that turns the original XML into the task data. Other benchmarks built on restricted text can use the same approach.
  • I3. Strong classical baselines: a well-tuned feature-based model was released next to the neural entries. This showed how much deep models actually add, which was less than one might expect, and it is a useful sanity check for any new benchmark.
  • I4. Subset breakdowns with rank correlations: the authors reported both the score spread across POS and genre subsets and the rank correlation with overall results. Together these show whether a leaderboard ranking is stable or depends on one subset.
  • I5. Relating difficulty to label prevalence: per-POS performance was compared with the share of positive labels in each POS. This is an easy way to separate class-imbalance effects from real linguistic difficulty in token-level tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Idioms & multiword expressions

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands71 works · 79,805 citations🤝Educational Testing ServicePrinceton · United States12 works · 1,801 citations

Cite

Cite as Chee Wee (Ben) Leong, Beata Beigman Klebanov and Ekaterina Shutova. 2018. A Report on the 2018 VUA Metaphor Detection Shared Task. Proceedings of the Workshop on Figurative Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-0907

@inproceedings{leong2018report,
  title = {{A Report on the 2018 VUA Metaphor Detection Shared Task}},
  author = {Chee Wee (Ben) Leong, Beata Beigman Klebanov and Ekaterina Shutova},
  booktitle = {Proceedings of the Workshop on Figurative Language Processing},
  year = {2018},
  doi = {10.18653/v1/w18-0907},
  url = {https://doi.org/10.18653/v1/w18-0907}
}

References

49 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Beata Beigman Klebanov, Daniel Diermeier, and Eyal Beigman. 2008. Lexical cohesion analysis of political speech. Political Analysis, 16(4):447–463. find
  2. Beata Beigman Klebanov, Chee Wee Leong, and Michael Flor. 2018. A corpus of non-native written english annotated for metaphor. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics, New Orleans,LA. find
  3. Beata Beigman Klebanov, Chee Wee Leong, E Dario Gutierrez, Ekaterina Shutova, and Michael Flor. 2016. Semantic classifications for detection of verb E Dario Gutierrez, Guillermo Cecchi, Cheryl Corcoran, and Philip Corlett. 2017. Using automated metaphor identification to aid in detection and predict… find
  4. Beata Beigman Klebanov, Chee Wee Leong, Michael Heilman, and Michael Flor. 2014. Different texts, same metaphors: Unigrams and beyond. In Proceedings of the Second Workshop on Metaphor in NLP, pages 11–17. find
  5. E Dario Gutierrez, Ekaterina Shutova, Tyler Marghetis, and Benjamin Bergen. 2016. Literal and metaphorical senses in compositional distributional semantic models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 183–1… find
  6. Richard M Billow, Jeffrey Rossman, Nona Lewis, De-berah Goldman, and Charles Raps. 1997. Observing expressive and deviant language in schizophrenia. Metaphor and Symbol, 12(3):205–216. find
  7. Steven Bird and Edward Loper. 2004. Nltk: the natural language toolkit. In Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, page 31. Association for Computational Linguistics. our card
  8. Dirk Hovy, Shashank Shrivastava, Sujay Kumar Jauhar, Mrinmaya Sachan, Kartik Goyal, Huying Li, Whitney Sanders, and Eduard Hovy. 2013. Identifying metaphorical word use with tree kernels. In Proceedings of the First Workshop on Metaphor in NLP, pages 52–57. find
  9. Julia Birke and Anoop Sarkar. 2006. A clustering approach for nearly unsupervised recognition of non-literal language. In 11th Conference of the European Chapter of the Association for Computational Linguistics. find
  10. Hyeju Jang, Seungwhan Moon, Yohan Jo, and Carolyn Rose. 2015. Metaphor detection in discourse. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 384–392. find
  11. Yuri Bizzoni and Mehdi Ghanimifard. 2018. Bigrams and bilstms: Two neural networks for sequential metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, New Orleans,LA. find
  12. Sujata S Kathpalia and Heah Lee Hah Carmel. 2011. Metaphorical competence in esl student writing. RELC Journal, 42(3):273–290. find
  13. Hossein Kaviani and Robabeh Hamedi. 2011. A quantitative/qualitative study on metaphors used by persian depressed patients. Archives of Psychiatry and Psychotherapy, 4(5-13):110. find
  14. Jan Bosman. 1987. Persuasive effects of political metaphors. Metaphor and Symbol, 2(2):97–113. find
  15. Luana Bulat, Stephen Clark, and Ekaterina Shutova. 2017. Modelling metaphor with attribute-based semantics. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 523–528. find
  16. Maximilian Köper and Sabine Schulte im Walde. 2017. Improving verb metaphor detection by propagating abstractness to words, phrases and individual senses. In Proceedings of the 1st Workshop on Sense, Concept and Entity Representations and their Applications, pages 24–30. find
  17. Lynne Cameron. 2003. Metaphor in educational discourse. A&C Black. find
  18. George Lakoff. 2010. Moral politics: How liberals and conservatives think. University of Chicago Press. find
  19. Erik-Lân Do Dinh and Iryna Gurevych. 2016. Token-level metaphor detection using neural networks. In Proceedings of the Fourth Workshop on Metaphor in NLP, pages 28–33. find
  20. George Lakoff and Mark Johnson. 2008. Metaphors we live by. University of Chicago press. find
  21. Mark J Landau, Daniel Sullivan, and Jeff Greenberg. 2009. Evidence that self-relevant motives and metaphoric framing interact to influence political and social attitudes. Psychological Science, 20(11):1421–1427. find
  22. Jonathan Dunn. 2013. What metaphor identification systems can tell us about metaphor-in-language. In Proceedings of the First Workshop on Metaphor in NLP, pages 1–10. find
  23. Jeannette Littlemore, Tina Krennmayr, James Turner, and Sarah Turner. 2013. An investigation into metaphor use at different levels of second language writing. Applied linguistics, 35(2):117–144. find
  24. Matt Gedigian, John Bryant, Srini Narayanan, and Branimir Ciric. 2006. Catching metaphors. In Proceedings of the Third Workshop on Scalable Natural Language Understanding, pages 41–48. Association for Computational Linguistics. find
  25. Jeannette Littlemore and Graham Low. 2006. Metaphoric competence, second language learning, and communicative language ability. Applied linguistics, 27(2):268–294. find
  26. Pragglejaz Group. 2007. Mip: A method for identifying metaphorically used words in discourse. Metaphor and symbol, 22(1):1–39. find
  27. Filip Skurniak, Maria Janicka, and Aleksander Wawer. 2018. Multim-module recurrent neural networks with transfer learning. a submission for the metaphor detection shared task. In Proceedings of the Workshop on Figurative Language Processing, New Orleans,LA. find
  28. Saif Mohammad, Ekaterina Shutova, and Peter Turney. 2016. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 23–33. our card
  29. Michael Mohler, David Bracewell, Marc Tomlinson, and David Hinote. 2013. Semantic signatures for example-based linguistic metaphor detection. In Proceedings of the First Workshop on Metaphor in NLP, pages 27–35. find
  30. Gerard J Steen, Aletta G Dorst, J Berenike Herrmann, Anna Kaal, Tina Krennmayr, and Trijntje Pasma. 2010. A method for linguistic metaphor identification: From MIP to MIPVU, volume 14. John Benjamins Publishing. find
  31. Anna Mosolova, Ivan Bondarenko, and Vadim Fomin. 2018. Conditional random fields for metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, New Orleans,LA. find
  32. Egon Stemle and Alexander Onysko. 2018. Using language learner data for metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, New Orleans,LA. find
  33. Agnieszka Mykowiecka, Aleksander Wawer, and Malgorzata Marciniak. 2018. Detecting figurative word occurrences using word embeddings. In Proceedings of the Workshop on Figurative Language Processing, New Orleans,LA. find
  34. Krishnkant Swarnkar and Anil Kumar Singh. 2018. Di-lstm contrast : A deep neural network for metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, New Orleans,LA. find
  35. Yair Neuman, Dan Assaf, Yohai Cohen, Mark Last, Shlomo Argamon, Newton Howard, and Ophir Frieder. 2013. Metaphor identification in large texts corpora. PloS one, 8(4):e62343. find
  36. Serra Sinem Tekiroglu, Gözde Özbal, and Carlo Strapparava. 2015. Exploring sensorial features for metaphor identification. In Proceedings of the Third Workshop on Metaphor in NLP, pages 31–39. find
  37. Gözde Özbal, Carlo Strapparava, and Serra Sinem Tekiroglu. 2016. Prometheus: A corpus of proverbs annotated with metaphors. In LREC. find
  38. Paul H Thibodeau and Lera Boroditsky. 2011. Metaphors we think with: The role of metaphor in reasoning. PloS one, 6(2):e16782. find
  39. Malay Pramanick, Ashim Gupta, and Pabitra Mitra. 2018. An lstm-crf based approach to token-level metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, New Orleans,LA. find
  40. Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 248–258. our card
  41. Marek Rei, Luana Bulat, Douwe Kiela, and Ekaterina Shutova. 2017. Grasping the finer point: A supervised similarity network for metaphor detection. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1537–1546. find
  42. Yulia Tsvetkov, Elena Mukomel, and Anatole Gershman. 2013. Cross-lingual metaphor detection using common semantic features. In Proceedings of the First Workshop on Metaphor in NLP, pages 45–51. find
  43. Ekaterina Shutova, Douwe Kiela, and Jean Maillard. 2016. Black holes and white rabbits: Metaphor identification with visual features. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 160–170. our card
  44. Peter D Turney, Yair Neuman, Dan Assaf, and Yohai Cohen. 2011. Literal and metaphorical sense identification through concrete and abstract context. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 680– 690. Association for Computational Linguistics. our card
  45. Ekaterina Shutova, Lin Sun, Elkin Darı́o Gutiérrez, Patricia Lichtenstein, and Srini Narayanan. 2017. Multilingual metaphor processing: Experiments with semi-supervised and unsupervised learning. Computational Linguistics, 43(1):71–123. find
  46. Tony Veale, Ekaterina Shutova, and Beata Beigman Klebanov. 2016. Metaphor: A computational perspective. Synthesis Lectures on Human Language Technologies, 9(1):1–160. find
  47. Ekaterina Shutova, Lin Sun, and Anna Korhonen. 2010. Metaphor identification using verb and noun clustering. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1002–1010. Association for Computational Linguistics. our card
  48. Chuhan Wu, Fangzhao Wu, Yubo Chen, Sixing Wu, Zhigang Yuan, and Yongfeng Huang. 2018. Thu ngn at naacl-2018 metaphor shared task: Neural metaphor detecting with cnn-lstm model. In Proceedings of the Workshop on Figurative Language Processing, New Orleans,LA. find
  49. Gerald Zaltman and Lindsay H Zaltman. 2008. Marketing metaphoria: What deep metaphors reveal about the minds of consumers. Harvard Business Press. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/leong-2018-report-2018-vua-metaphor-detection/