Elamur Research⌕
☰
home / papers / 2021 / Scalar Adjective Identification and Multilingual Ranking

Scalar Adjective Identification and Multilingual Ranking

Paper · 2021 · Aina Garí Soler and Marianna Apidianaki

Summary

Most earlier work on ordering scalar adjectives by intensity (for example, nice < great < wonderful) used English only, because evaluation data existed only for English. It also relied mostly on lexical patterns and lexicons. The authors release two benchmarks.

  • MULTI-SCALE is the DE MELO and WILKINSON scale datasets translated into French, Spanish and Greek. Each scale comes with OSCAR sentences in which every adjective of the scale appears in the same ten contexts.
  • SCAL-REL is an English dataset that balances scalar adjectives against relational (denominal) adjectives, posing a new identification task.

For ranking, they test multilingual BERT, four monolingual BERT-style models and fastText. The main method is DIFFVEC: it builds an intensity direction from the representations of a mild/extreme adjective pair and orders adjectives by their cosine to that direction. A translated perfect–good pair alone gives competitive results in every language. Frequency and polysemy baselines also perform well, though scores stay below English levels.

On SCAL-REL, logistic regression over averaged BERT vectors reaches 0.946 accuracy. The work matters for inference, sentiment and emotion analysis beyond English.

Key points

  • MULTI-SCALE is built from 87 DE MELO half-scales and 21 WILKINSON half-scales (12 full scales, split as in Cocos et al.). Translators saw each adjective inside its scale, so a polysemous word like hot became Spanish caliente or picante depending on the scale.
  • Context sentences were made by drawing ten OSCAR sentences per scale and then substituting each scale member into every sentence. English uses the ukWaC-Random set.
  • SCAL-REL pairs 443 scalar adjectives (from DE MELO, WILKINSON and CROWD) with 443 WordNet pertainyms. The pertainyms are frequency-balanced: 222 frequent, 221 rare. The data is split 65/10/25.
  • On ranking, the best BERT layer differs by language: the French and Greek monolingual models peak in early layers, while others peak in layers 6–12. On Greek DE MELO, FREQ and fastText beat the contextual models. French scores are lower, likely because it has more ties (58% of French DE MELO scales contain one, versus 45% in English).
  • On SCAL-REL, BERT ADJ-REP scores 0.946 and fastText 0.929, while the single-feature DIFFVEC cosine lags well behind. Remaining errors are borderline words such as microscopic, sympathetic and imperative.

Interesting ideas

  • I1. An intensity axis can be obtained by subtracting a mild adjective's vector from an extreme one's in identical sentences; adjectives on the same scale are then sorted by cosine to that axis, needing almost no labelled data.
  • I2. The absolute cosine to the intensity direction serves as a "does intensity apply at all" score: vectors roughly perpendicular to it indicate non-gradable, classificatory adjectives.
  • I3. Placing every candidate word into the exact same set of sentences removes context as a confound when comparing words.
  • I4. Milder adjectives tend to be more frequent and more polysemous than extreme ones, so raw frequency or sense counts give a surprisingly strong ordering heuristic across languages.
  • I5. Translating a word while showing its whole scale pins down which sense is meant, so the translation stays faithful to the scalar meaning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): This could be used offline to pre-order adjectives within each WordNet-derived scale in the compiled knowledge, so entailments like "wonderful ⇒ good but not the reverse" become explicit, explainable edges.
  • I2 (world): This could tag adjectives as gradable or relational in the ontology, so comparatives and degree modifiers ("very wooden") are only built for adjectives where a scale exists.
  • I3 (learn): The same substitution trick can build controlled evaluation sets where one lexical item is swapped, acting as a negative control for whether a parser or frame extractor reacts to the word itself rather than its surroundings.
  • I4 (world): This gives a cheap, transparent fallback for ordering adjectives on a scale when no curated intensity data exists, clearly labelled as a heuristic in the explanation trace.
  • I5 (back): During translation through the meta-world, the scale or frame an adjective belongs to could select the target word, for example choosing picante rather than caliente for spicy-hot.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France104 works · 60,877 citations🎓University of HelsinkiHelsinki · Finland33 works · 4,705 citations🎓Université Paris-SaclayGif-sur-Yvette · France34 works · 5,733 citations🔬Laboratoire Interdisciplinaire des Sciences du NumériqueGif-sur-Yvette · France10 works · 422 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksMultilingual & cross-lingual NLP · 465 works

Cite

Cite as Aina Garí Soler and Marianna Apidianaki. 2021. Scalar Adjective Identification and Multilingual Ranking. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.370

@inproceedings{soler2021scalar,
  title = {{Scalar Adjective Identification and Multilingual Ranking}},
  author = {Aina Garí Soler and Marianna Apidianaki},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.370},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.370}
}

References

20 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. Journal of Language Resources and Evaluation, 43(3):209–226. ↗
  2. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In Advances in Neural Information Processing Systems 29, pages 4349–4357. Barcelona, Spain.
  3. Thorsten Brants and Alex Franz. 2006. Web 1T 5-gram Version 1. In LDC2006T13, Philadelphia, Pennsylvania. Linguistic Data Consortium. find
  4. José Cañete, Gabriel Chaperon, Rodrigo Fuentes, and Jorge Pérez. 2020. Spanish Pre-Trained BERT Model and Evaluation Data. In PML4DC at ICLR 2020. find
  5. Anne Cocos, Skyler Wharton, Ellie Pavlick, Marianna Apidianaki, and Chris Callison-Burch. 2018. Learning Scalar Adjective Intensity from Paraphrases. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1752–1762, Brussels, Belgium. Association for Computational Linguistics.
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7371–7385, Online. Association for Computational Linguistics.
  7. Bart Geurts. 2010. Quantity implicatures. Cambridge University Press. find
  8. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  9. John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2020. GREEK-BERT: The Greeks visiting Sesame Street. In 11th Hellenic Conference on Artificial Intelligence, pages 110–117. find
  10. Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised Language Model Pre-training for French. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 2479–2490, Marseille, France. European Language Resources Association. ↗
  11. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2010. “Was It Good? It Was Provocative.” Learning the Meaning of Scalar Adjectives". In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 167–176, Uppsala, Sweden. Association for Computational Linguistics.
  12. Louise McNally and Gemma Boleda. 2004. Relational adjectives as properties of kinds. Colloque de Syntaxe et Sémantique à Paris. find
  13. Gerard de Melo and Mohit Bansal. 2013. Good, Great, Excellent: Global Inference of Semantic Intensities. Transactions of the Association for Computational Linguistics, 1:279–290. ↗
  14. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250. ↗
  15. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. Language, Speech, and Communication. MIT Press, Cambridge, MA. find
  16. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. find
  17. Aina Garí Soler and Marianna Apidianaki. 2020. BERT knows punta cana is not just beautiful, it’s gorgeous: Ranking scalar adjectives with contextualised representations. In Proceedings of the 2020
  18. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational Linguistics, 37(2):267–307. ↗
  19. Bryan Wilkinson and Tim Oates. 2016. A Gold Standard for Scalar Adjectives. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2669– 2675, Portorož, Slovenia. European Language Resources Association (ELRA).
  20. George Kingsley Zipf. 1945. The meaning-frequency relationship of words. Journal of General Psychology, 33(2):251–256. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/soler-2021-scalar-adjective-identification-multilingual-ranking/