Elamur Research⌕
☰
home / papers / 2018 / Do Character-Level Neural Network Language Models Capture Knowledge of Multiword Expression Compositionality?

Do Character-Level Neural Network Language Models Capture Knowledge of Multiword Expression Compositionality?

Paper · 2018 · Ali Hakimi Parizi and Paul Cook

Summary

The paper asks whether character-level recurrent language models encode how compositional a multiword expression (MWE) is, that is, how much its meaning follows from its parts. The authors train a character-level LSTM language model on roughly 100M-character samples of English and German Wikipedia. They feed each MWE, and each of its component words, through the model as a plain character string. The final hidden state serves as the vector. Compositionality is then scored with cosine similarity, in the style of Salehi et al. (2015): either a weighted mix of MWE–component similarities (comp1) or similarity to the sum of the component vectors (comp2). They evaluate on English noun compounds (ENC), English verb-particle constructions (EVPC) and German noun compounds (GNC).

Correlations with human ratings are modest but significant for ENC and for the particle side of EVPC. They are not significant for GNC or for the verb side of EVPC. All results stay well below word2vec-based baselines. The main appeal is practical. The method does not need MWE occurrences to be identified in a corpus, and it can score rare or unseen expressions.

Key points

  • Default model: a TensorFlow char-RNN with a two-layer LSTM, one-hot characters and 128 hidden units (batch 20, learning rate 0.002, no dropout). Training data were 1% of the English and 2% of the German Wikipedia dumps (20 Jan 2018), cleaned with WP2TXT.
  • Evaluation sets: ENC has 90 compounds (Reddy et al., 2011), EVPC has 160 verb-particle constructions (Bannard, 2006) and GNC has 244 compounds (von der Heide and Borgwaldt, 2009). Scores are Pearson's r against human ratings, with α = 0.7 for the noun-compound sets and 0.5 for EVPC.
  • Pearson's r for comp1 and comp2:
    • ENC: 0.239 and 0.286, both significant, compared with 0.717 for the word2vec baseline.
    • EVPC particle: 0.313 and 0.301, both significant.
    • EVPC verb and GNC: near zero or negative.
  • On the 71 EVPC items that occur fewer than 5 times in training, particle correlations remain significant (0.327 and 0.308). Unseen English noun compounds, however, receive no significant prediction.
  • Several variants gave no consistent gain over the default setup: a backward LSTM, a bi-directional LSTM, a plain RNN, a GRU, learned character embeddings of size 25 or 50, a larger batch, and dropout.

Interesting ideas

  • I1. An expression's vector can be read directly from a character model's final state by spelling out the whole phrase, space included, which avoids corpus-level detection of that phrase.
  • I2. Scoring each component's contribution separately, instead of one overall number, shows which part of a phrase keeps its literal meaning (here, the particle more than the verb).
  • I3. Splitting the evaluation into attested, rare and unseen items shows where a model truly generalises and where it merely remembers.
  • I4. The authors flag two practical limits: "gappy" expressions (with a slot inside) cannot simply be spelled out, and split verb-particle constructions had to be fed to the model in joined form.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Phrase vectors could be built this way for rare compounds missing from WordNet, giving a fallback signal on whether the compound should inherit its head's ontology class.
  • I2 (frames): When mapping phrasal verbs to VerbNet frames, a per-component literalness estimate could decide whether the particle adds a directional or path role or whether the whole verb should be treated as a single lexical unit.
  • I3 (learn): Evaluation reports could routinely break scores down by training frequency, used as a control alongside negative controls, so that gains on frequent items do not mask failure on unseen ones.
  • I4 (tree): After UD parsing, compound:prt and similar relations could be used to rejoin split particles into a canonical string before any lexicon or compositionality lookup.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of New BrunswickFredericton · Canada5 works · 97 citations

Also on the byline: Razi University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksIdioms & multiword expressions · 222 works

Cite

Cite as Ali Hakimi Parizi and Paul Cook. 2018. Do Character-Level Neural Network Language Models Capture Knowledge of Multiword Expression Compositionality?. Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{parizi2018do,
  title = {{Do Character-Level Neural Network Language Models Capture Knowledge of Multiword Expression Compositionality?}},
  author = {Ali Hakimi Parizi and Paul Cook},
  booktitle = {Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018)},
  year = {2018},
  url = {https://aclanthology.org/W18-4920/}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Timothy Baldwin, Colin Bannard, Takaaki Tanaka, and Dominic Widdows. 2003. An empirical model of multiword expression decomposability. In Proceedings of the ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment. Association for Computational Linguistics, Sapporo, Japan, pages 89–96.
  2. Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. Handbook of natural language processing 2:267–292. find
  3. Colin James Bannard. 2006. Acquiring phrasal lexicons from corpora. Ph.D. thesis, University of Edinburgh. find
  4. Thorsten Brants, Ashok C Popat, Peng Xu, Franz J Och, and Jeffrey Dean. 2007. Large language models in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).
  5. Marine Carpuat and Mona Diab. 2010. Task-based evaluation of multiword expressions: a pilot study in statistical machine translation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Los Angeles, California, pages 242–245. their training corpus. find
  6. Yi Chen, Ming Zhou, and Shilong Wang. 2006. Reranking answers for definitional qa using language modeling. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 1081–1088. find
  7. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 . ↗
  8. Michael Collins, Brian Roark, and Murat Saraclar. 2005. Discriminative syntactic language modeling for speech recognition. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 507–514. find
  9. Leonardo dos Santos Pinheiro and Mark Dras. 2017. Stock market prediction with deep learning: A character-based neural language model for event-based trading. In Proceedings of the Australasian Language Technology Association Workshop 2017. pages 6–15. find
  10. Waseem Gharbieh, Virendrakumar Bhavsar, and Paul Cook. 2017. Deep learning models for multiword expression identification. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017). Association for Computational Linguistics, Vancouver, Canada, pages 54– 64. find
  11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. 9(8):1735–1780. Long short-term memory. Neural computation Graham Katz and Eugenie Giesbrecht. 2006. Automatic identification of non-compositional multi-word expressions using latent semantic analysis. In Proceedings of the Workshop on Multiword Expressions: Identifying and Exploiting Underlying Properties. Sydney, Australia, pages 12–19.
  12. Su Nam Kim and Timothy Baldwin. 2010. How to pick out token instances of english verb-particle constructions. Language Resources and Evaluation 44(1–2):97–113. ↗
  13. Ioannis Korkontzelos and Suresh Manandhar. 2010. Can recognising multiword expressions improve shallow parsing? In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Los Angeles, California, pages 636–644. find
  14. Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1520–1530. ↗
  15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of Workshop at the International Conference on Learning Representations, 2013. Scottsdale, USA. ↗
  16. Tomáš Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Cernocky. 2012. Subword language modeling with neural networks. preprint (http://www.fit. vutbr. cz/imikolov/rnnlm/char. pdf) . ↗
  17. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive science 34(8):1388–1429. ↗
  18. Fuchun Peng, Dale Schuurmans, Shaojun Wang, and Vlado Keselj. 2003. Language independent authorship attribution using character level language models. In Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics-Volume 1. Association for Computational Linguistics, pages 267–274. find
  19. Siva Reddy, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study on compositionality in compound nouns. In Proceedings of 5th International Joint Conference on Natural Language Processing. pages 210–218.
  20. Bahar Salehi, Paul Cook, and Timothy Baldwin. 2014. Using distributional similarity of multi-way translations to predict multiword expression compositionality. In Proceedings of the 14th Conference of the EACL (EACL 2014). Gothenburg, Sweden, pages 472–481. find
  21. Bahar Salehi, Paul Cook, and Timothy Baldwin. 2015. A word embedding approach to predicting the compositionality of multiword expressions. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Denver, Colorado, pages 977–983. find
  22. Giancarlo Salton, Robert Ross, and John Kelleher. 2016. Idiom token classification using sentential distributed semantics. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 194–204.
  23. Cicero D Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31st International Conference on Machine Learning (ICML-14). pages 1818–1826. find
  24. Nathan Schneider, Emily Danchik, Chris Dyer, and Noah A. Smith. 2014. Discriminative lexical semantic segmentation with gaps: Running the mwe gamut. Transactions of the Association of Computational Linguistics 2:193–206. ↗
  25. Patrick Schone and Daniel Jurafsky. 2001. Is knowledge-free induction of multiword unit dictionary headwords a solved problem? In Proceedings of the 6th Conference on Empirical Methods in Natural Language Processing (EMNLP 2001). Hong Kong, China, pages 100–108. find
  26. Sabine Schulte im Walde, Stefan Müller, and Stefan Roller. 2013. Exploring vector space models to predict the compositionality of German noun-noun compounds. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity. Atlanta, GA, pages 255–265. find
  27. Raymond Hendy Susanto, Hai Leong Chieu, and Wei Lu. 2016. Learning to capitalize with character-level recurrent neural networks: An empirical study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pages 2090–2095. find
  28. Claudia von der Heide and Susanne Borgwaldt. 2009. Assoziationen zu unter-, basis-und oberbegriffen. eine explorative studie. In Proceedings of the 9th Norddeutsches Linguistisches Kolloquium. pages 51–74. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/parizi-2018-do-character-level-neural-network/