Elamur Research⌕
☰
home / papers / 2020 / A Report on the 2020 VUA and TOEFL Metaphor Detection Shared Task

A Report on the 2020 VUA and TOEFL Metaphor Detection Shared Task

Summary

This report covers the second shared task on token-level metaphor detection, held at the ACL 2020 Workshop on Figurative Language Processing as a follow-up to the 2018 edition. Systems had to label every content word (nouns, verbs, adjectives, adverbs, excluding forms of be/do/have) as metaphorical or literal. There were four tracks: All POS and Verbs-only, each run on the VU Amsterdam Metaphor Corpus (VUA) and on a metaphor-annotated subset of the TOEFL learner-essay corpus. The organizers released the data, scripts and features, and provided three baselines: a feature-based classifier, the 2018 bot.zen BiLSTM, and fine-tuned BERT. Submissions were scored on CodaLab by F1 for the metaphor class. Thirteen teams wrote system papers, and most relied on contextual embeddings such as BERT, RoBERTa and ELMo. Some teams combined these with linguistic features, ensembles or Transformer stacks. The authors also broke results down by proficiency level and part of speech. For VUA, these breakdowns reproduced the 2018 patterns, while learner essays behaved quite differently. The task establishes a reusable benchmark for metaphor detection on both native and non-native text.

Key points

  • VUA data: 117 BNC fragments from four genres (Academic, News, Conversation, Fiction), annotated with MIPVU at κ > 0.8. Metaphor rates are 29% for Verbs and 18% for All POS.
  • TOEFL data: 240 essays (180 train / 60 test) taken from the ETS Corpus of Non-Native Written English. They cover 8 prompts, 3 native languages (Japanese, Italian, Arabic) and 2 proficiency levels. Annotation targeted argumentation-relevant metaphors with κ = 0.56–0.62. Metaphor rates are only 13% for Verbs and 7% for All POS.
  • Participation: VUA All POS drew 13 teams and 210 submissions; TOEFL All POS drew 9 teams and 247 submissions. Teams could submit at most 5 times per day per track and up to 12 final systems.
  • Best All-POS F1 in the per-POS breakdown: DeepMet scored .769 on VUA and .715 on TOEFL. The BERT baseline reached .718 on VUA and .624 on TOEFL.
  • POS patterns:
    • On VUA, Verbs were easiest and Adverbs hardest, with an average gap of about 11 F1 points.
    • On TOEFL, Adjectives usually came out on top and Nouns were hardest for every system, with an average gap of 17 points.
    • The best TOEFL Nouns score was just 0.641.

Interesting ideas

  • I1. Break results down by subgroup, not just overall. Slicing scores by proficiency level and by part of speech exposes where systems fail. For example, nominal metaphors turned out to be rare and hard in learner text. Any shared task or benchmark would benefit from such breakdowns.
  • I2. Train on the broader task, then evaluate on the narrower one. The BERT baseline for the Verbs tracks was tuned on All POS data and tested on verbs, which worked better than training on verbs alone. This can help whenever a sub-task has too little data.
  • I3. Combine pretrained embeddings with hand-crafted lexical features. illiniMet concatenated RoBERTa vectors with WordNet, VerbNet and concreteness features, then took a majority vote over models trained on different data splits. This is a cheap way to inject lexical knowledge and reduce variance.
  • I4. Frame token tagging as reading comprehension. DeepMet treated each word as a query and fed in global context, local context and POS. Its ablations showed that fine-grained POS and global context mattered most. This query-style framing could carry over to other word-level semantic tagging tasks.
  • I5. Check the effect of the prompt or domain. Prompts that themselves suggest a metaphor inflate metaphor frequency in the essays written for them. It is worth controlling for task-induced data patterns when collecting and evaluating any learner or crowd corpus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Idioms & multiword expressions

Cite

Cite as Chee Wee (Ben) Leong, Beata Beigman Klebanov, Chris Hamill, Egon Stemle, Rutuja Ubale and Xianyang Chen. 2020. A Report on the 2020 VUA and TOEFL Metaphor Detection Shared Task. Proceedings of the Second Workshop on Figurative Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.figlang-1.3

@inproceedings{leong2020report,
  title = {{A Report on the 2020 VUA and TOEFL Metaphor Detection Shared Task}},
  author = {Chee Wee (Ben) Leong, Beata Beigman Klebanov, Chris Hamill, Egon Stemle, Rutuja Ubale and Xianyang Chen},
  booktitle = {Proceedings of the Second Workshop on Figurative Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.figlang-1.3},
  url = {https://doi.org/10.18653/v1/2020.figlang-1.3}
}

References

64 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ghadi Alnafesah, Harish Tayyar Madabushi, and Mark Lee. 2020. Augmenting neural metaphor detection with concreteness. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  2. Beata Beigman Klebanov, Daniel Diermeier, and Eyal Beigman. 2008. Lexical cohesion analysis of political speech. Political Analysis, 16(4):447–463. find
  3. Beata Beigman Klebanov and Michael Flor. 2013. Argumentation-relevant metaphors in test-taker essays. In Proceedings of the First Workshop on Metaphor in NLP, pages 11–20. find
  4. Beata Beigman Klebanov, Chee Wee Leong, and Michael Flor. 2018. A corpus of non-native written English annotated for metaphor. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), p… find
  5. Beata Beigman Klebanov, Chee Wee Leong, E Dario Gutierrez, Ekaterina Shutova, and Michael Flor. 2016. Semantic classifications for detection of verb metaphors. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 101–106… find
  6. Beata Beigman Klebanov, Chee Wee Leong, Michael Heilman, and Michael Flor. 2014. Different texts, same metaphors: Unigrams and beyond. In Proceedings of the Second Workshop on Metaphor in NLP, pages 11–17. find
  7. Richard M Billow, Jeffrey Rossman, Nona Lewis, De-berah Goldman, and Charles Raps. 1997. Observing expressive and deviant language in schizophrenia. Metaphor and Symbol, 12(3):205–216. find
  8. Julia Birke and Anoop Sarkar. 2006. A clustering approach for nearly unsupervised recognition of non-literal language. In 11th Conference of the European Chapter of the Association for Computational Linguistics. find
  9. Lynne Cameron. 2003. Metaphor in educational discourse. A&C Black. find
  10. Xianyang Chen, Chee Wee Leong, Michael Flor, and Beata Beigman Klebanov. 2020. Go figure! multi-task transformer-based architecture for metaphor detection using idioms: Ets team in 2020 metaphor shared task. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  11. Verna Dankers, Karan Malhotra, Gaurav Kudva, Volodymyr Medentsiy, and Ekaterina Shutova. 2020. Being neighbourly: Neural metaphor identification in discourse. In Proceedings of the Second Workshop on Figurative Language Processing, pages 227–234, Online. Association for Computational Linguistics. find
  12. Verna Dankers, Marek Rei, Martha Lewis, and Ekaterina Shutova. 2019. Modelling the interplay of metaphor and emotion through multitask learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pr… find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. our card
  14. Erik-Lân Do Dinh and Iryna Gurevych. 2016. Token-level metaphor detection using neural networks. In Proceedings of the Fourth Workshop on Metaphor in NLP, pages 28–33. find
  15. Jonathan Dunn. 2013. What metaphor identification systems can tell us about metaphor-in-language. In Proceedings of the First Workshop on Metaphor in NLP, pages 1–10. find
  16. Ge Gao, Eunsol Choi, Yejin Choi, and Luke Zettlemoyer. 2018. Neural metaphor detection in context. In Conference on Empirical Methods in Natural Language Processing. our card
  17. Jan Bosman. 1987. Persuasive effects of political metaphors. Metaphor and Symbol, 2(2):97–113. find
  18. Matt Gedigian, John Bryant, Srini Narayanan, and Branimir Ciric. 2006. Catching metaphors. In Proceedings of the Third Workshop on Scalable Natural Language Understanding, pages 41–48. Association for Computational Linguistics. find
  19. Jennifer Brooks and Abdou Youssef. 2020. Metaphor detection using ensembles of bidirectional recurrent neural networks. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  20. Hongyu Gong, Kshitij Gupta, Akriti Jain, and Suma Bhat. 2020. Illinimet: Illinois system for metaphor detection with contextual and linguistic information. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  21. Luana Bulat, Stephen Clark, and Ekaterina Shutova. 2017. Modelling metaphor with attribute-based semantics. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 523–528. find
  22. E Dario Gutierrez, Guillermo Cecchi, Cheryl Corcoran, and Philip Corlett. 2017. Using automated metaphor identification to aid in detection and prediction of first-episode schizophrenia. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2923–2930. find
  23. E Dario Gutierrez, Ekaterina Shutova, Tyler Marghetis, and Benjamin Bergen. 2016. Literal and metaphorical senses in compositional distributional semantic models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 183–1… find
  24. Dirk Hovy, Shashank Shrivastava, Sujay Kumar Jauhar, Mrinmaya Sachan, Kartik Goyal, Huying Li, Whitney Sanders, and Eduard Hovy. 2013. Identifying metaphorical word use with tree kernels. In Proceedings of the First Workshop on Metaphor in NLP, pages 52–57. find
  25. Chee Wee Leong, Beata Beigman Klebanov, and Ekaterina Shutova. 2018. A report on the 2018 vua metaphor detection shared task. In Proceedings of the Workshop on Figurative Language Processing, pages 56–66. our card
  26. Hyeju Jang, Seungwhan Moon, Yohan Jo, and Carolyn Rose. 2015. Metaphor detection in discourse. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 384–392. find
  27. Sujata S Kathpalia and Heah Lee Hah Carmel. 2011. Metaphorical competence in esl student writing. RELC Journal, 42(3):273–290. find
  28. Hossein Kaviani and Robabeh Hamedi. 2011. A quantitative/qualitative study on metaphors used by persian depressed patients. Archives of Psychiatry and Psychotherapy, 4(5-13):110. find
  29. Maximilian Köper and Sabine Schulte im Walde. 2017. Improving verb metaphor detection by propagating abstractness to words, phrases and individual senses. In Proceedings of the 1st Workshop on Sense, Concept and Entity Representations and their Applications, pages 24–30. find
  30. Tarun Kumar and Yashvardhan Sharma. 2020. Character aware models with similarity learning for metaphor detection. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  31. Kevin Kuo and Marine Carpuat. 2020. Evaluating a bilstm model for metaphor detection in toefl essays. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  32. George Lakoff. 2010. Moral politics: How liberals and conservatives think. University of Chicago Press. find
  33. George Lakoff and Mark Johnson. 2008. Metaphors we live by. University of Chicago press. find
  34. Mark J Landau, Daniel Sullivan, and Jeff Greenberg. 2009. Evidence that self-relevant motives and metaphoric framing interact to influence political and social attitudes. Psychological Science, 20(11):1421–1427. find
  35. Duong Le, My Thai, and Thien Nguyen. 2020. Multi-task learning for metaphor detection with graph convolutional neural networks and word sense disambiguation. In The Thirty-Fourth AAAI Conference Shuqun Li, Jingjie Zeng, Jinhui Zhang, Tao Peng, Liang Yang, and Hongfei Lin. 2020. Albert-BiLSTM for seq… find
  36. Jeannette Littlemore, Tina Krennmayr, James Turner, and Sarah Turner. 2013. An investigation into metaphor use at different levels of second language writing. Applied linguistics, 35(2):117–144. find
  37. Jeannette Littlemore and Graham Low. 2006. Metaphoric competence, second language learning, and communicative language ability. Applied linguistics, 27(2):268–294. find
  38. Jerry Liu, Nathan OH́ara, Alex Rubin, Rachel Draelos, and Cynthia Rudin. 2020. Metaphor detection using contextual word embeddings from transformers. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  39. Dermot Lynott, Louise Connell, Marc Brysbaert, James Brand, and James Carney. 2019. The Lancaster sensorimotor norms: multidimensional measures of perceptual and action strength for 40,000 English words. Behavior Research Methods, pages 1–21. find
  40. Rui Mao, Chenghua Lin, and Frank Guerin. 2019. End-to-end sequential metaphor identification inspired by linguistic theories. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3888–3898, Florence, Italy. Association for Computational Linguistics. find
  41. Rowan Hall Maudslay, Tiago Pimentel, Ryan Cotterell, and Simone Teufel. 2020. Metaphor detection using context and concreteness. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  42. Saif Mohammad, Ekaterina Shutova, and Peter Turney. 2016. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 23–33. our card
  43. Michael Mohler, David Bracewell, Marc Tomlinson, and David Hinote. 2013. Semantic signatures for example-based linguistic metaphor detection. In Proceedings of the First Workshop on Metaphor in NLP, pages 27–35. find
  44. Yair Neuman, Dan Assaf, Yohai Cohen, Mark Last, Shlomo Argamon, Newton Howard, and Ophir Frieder. 2013. Metaphor identification in large texts corpora. PloS one, 8(4):e62343. find
  45. Gözde Özbal, Carlo Strapparava, and Serra Sinem Tekiroglu. 2016. Prometheus: A corpus of proverbs annotated with metaphors. In LREC. find
  46. Marek Rei, Luana Bulat, Douwe Kiela, and Ekaterina Shutova. 2017. Grasping the finer point: A supervised similarity network for metaphor detection. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1537–1546. find
  47. Andrés Torres Rivera, Antoni Oliver, Salvador Climent, and Marta Coll-Florit. 2020. Neural metaphor detection with a residual bilstm-crf model. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  48. Ekaterina Shutova, Douwe Kiela, and Jean Maillard. 2016. Black holes and white rabbits: Metaphor identification with visual features. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 160–170. our card
  49. Ekaterina Shutova, Lin Sun, Elkin Darı́o Gutiérrez, Patricia Lichtenstein, and Srini Narayanan. 2017. Multilingual metaphor processing: Experiments with semi-supervised and unsupervised learning. Computational Linguistics, 43(1):71–123. find
  50. Ekaterina Shutova, Lin Sun, and Anna Korhonen. 2010. Metaphor identification using verb and noun clustering. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1002–1010. Association for Computational Linguistics. our card
  51. David Sidhu, Rachel Kwan, Penny Pexman, and Paul Siakaluk. 2014. Effects of relative embodiment in lexical and semantic processing of verbs. Acta psychologica, 149:32–39. find
  52. Gerard J Steen, Aletta G Dorst, J Berenike Herrmann, Anna Kaal, Tina Krennmayr, and Trijntje Pasma. 2010. A method for linguistic metaphor identification: From MIP to MIPVU, volume 14. John Benjamins Publishing. find
  53. Egon Stemle and Alexander Onysko. 2018. Using language learner data for metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, New Orleans,LA. find
  54. Egon Stemle and Alexander Onysko. 2020. Testing the role of metadata in metaphor identification. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  55. Chuandong Su, Fumiyo Fukumoto, Xiaoxi Huang, Jiyi Li, Rongbo Wang, and Zhiqun Chen. 2020. Deepmet: A reading comprehension paradigm for token-level metaphor detection. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  56. Serra Sinem Tekiroglu, Gözde Özbal, and Carlo Strapparava. 2015. Exploring sensorial features for metaphor identification. In Proceedings of the Third Workshop on Metaphor in NLP, pages 31–39. find
  57. Paul H Thibodeau and Lera Boroditsky. 2011. Metaphors we think with: The role of metaphor in reasoning. PloS one, 6(2):e16782. find
  58. Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 248–258. our card
  59. Yulia Tsvetkov, Elena Mukomel, and Anatole Gershman. 2013. Cross-lingual metaphor detection using common semantic features. In Proceedings of the First Workshop on Metaphor in NLP, pages 45–51. find
  60. Peter D Turney, Yair Neuman, Dan Assaf, and Yohai Cohen. 2011. Literal and metaphorical sense identification through concrete and abstract context. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 680– 690. Association for Computational Linguistics. our card
  61. Tony Veale, Ekaterina Shutova, and Beata Beigman Klebanov. 2016. Metaphor: A computational perspective. Synthesis Lectures on Human Language Technologies, 9(1):1–160. find
  62. Mingyu Wan, Kathleen Ahrens, Emmanuele Chersoni, Menghan Jiang, Qi Su, Rong Xiang, and Chu-Ren Huang. 2020. Using conceptual norms for metaphor detection. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA. find
  63. Chuhan Wu, Fangzhao Wu, Yubo Chen, Sixing Wu, Zhigang Yuan, and Yongfeng Huang. 2018. Thu ngn at naacl-2018 metaphor shared task: Neural metaphor detecting with cnn-lstm model. In Proceedings of the Workshop on Figurative Language Processing, New Orleans,LA. find
  64. Gerald Zaltman and Lindsay H Zaltman. 2008. Marketing metaphoria: What deep metaphors reveal about the minds of consumers. Harvard Business Press. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/leong-2020-report-2020-vua-toefl-metaphor/