Elamur Research⌕
☰
home / papers / 2024 / Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters

Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters

Paper · 2024 · Daniil Gurgurov, Mareike Hartmann and Simon Ostermann

Summary

Multilingual LLMs tend to underperform on languages with little training data. The authors test whether structured knowledge from ConceptNet can help, injected through adapters. ConceptNet is a multilingual graph that links concepts across languages. Eight low-resource languages are examined, namely Sinhala, Tibetan, Uyghur, Javanese, Nepali, Indonesian, Bulgarian and Maltese. For each one, the authors take that language's part of ConceptNet and rewrite its relations as plain-language predicates, producing short training sentences. They then train language adapters on this text, following the modular MAD-X design. Three objectives are compared: standard MLM, full-word masking and targeted masking. ConceptNet adapters are also set against Wikipedia-based adapters and an Adapter Fusion of both. All adapters are trained on data in the target language itself. The backbone is mBERT, and evaluation uses sentiment analysis and NER.

The results are mixed. Task adapters help fairly consistently. ConceptNet adapters mostly benefit sentiment analysis, not NER, and fusion seldom adds value. Gains still appear with small data and for languages missing from mBERT pre-training. This suggests cheap graph resources can be useful, but each language needs its own approach.

Key points

  • The pipeline stacks a language adapter (ConceptNet or Wikipedia) under a task adapter and a classification head, with optional Adapter Fusion to combine both knowledge sources.
  • ConceptNet triples are turned into text by mapping 12 relations to natural-language predicates. Examples are Antonym → "is the opposite of" and IsA → "is a type of".
  • ConceptNet adapters help sentiment analysis more often than NER. Wikipedia adapters likewise help SA more than NER, and fusion gave improvements in fewer cases.
  • Maltese, Uyghur, Tibetan and Sinhala are absent from mBERT pre-training, yet still benefited. This holds even though their ConceptNet and Wikipedia resources are much smaller than those for Bulgarian or Indonesian.
  • The authors name several limitations: the objectives may suit graph knowledge poorly, SA and NER may not need such knowledge, the language set is small, and larger backbones remain untested. Code is available on GitHub (d-gurgurov/Injecting-Commonsense-Knowledge-into-LLMs).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 worksKnowledge graphs & ontologies · 192 worksLow-resource languages · 267 works

Cite

Cite as Daniil Gurgurov, Mareike Hartmann and Simon Ostermann. 2024. Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters. arXiv:2407.01406 · 2024.kallm-1.7.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.kallm-1.7 · arXiv 2407.01406

@misc{gurgurov2024adapting,
  title = {{Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters}},
  author = {Daniil Gurgurov, Mareike Hartmann and Simon Ostermann},
  year = {2024},
  doi = {10.18653/v1/2024.kallm-1.7},
  eprint = {2407.01406},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.kallm-1.7}
}

References

39 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  3. Keith Cortis and Brian Davis. 2019. A social opinion gold standard for the Malta government budget 2018. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 364–369, Hong Kong, China. Association for Computational Linguistics. find
  4. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3504–3514.
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Alexiei Dingli and Nicole Sant. 2016. Sentiment analysis on maltese using machine learning. In Proceedings of The Tenth International Conference on Advances in Semantic Processing (SEMAPRO 2016), pages 21–25. find
  7. Viktor Hangya, Hossain Shaikh Saadi, and Alexander Fraser. 2022. Improving low-resource languages in pre-trained multilingual language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11993– 12006, Abu Dhabi, United Arab Emirates. Association f… find
  8. Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. 2021. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the…
  9. Yifan Hou, Wenxiang Jiao, Meizhen Liu, Carl Allen, Zhaopeng Tu, and Mrinmaya Sachan. 2022. Adapters for enhanced modeling of multilingual knowledge and text. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3902–3917. find
  10. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. find
  11. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
  12. Josip Jukić and Jan Snajder. 2023. Parameter-efficient language model tuning with active learning in low-resource settings. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5061–5074, Singapore. Association for Computational Linguistics. find
  13. Anne Lauscher, Olga Majewska, Leonardo F. R. Ribeiro, Iryna Gurevych, Nikolai Rozanov, and Goran Glavaš. 2020. Common sense or world knowledge? investigating adapter-based knowledge injection into pretrained transformers. In Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Kno… find
  14. Siyu Li, Kui Zhao, Jin Yang, Xinyun Jiang, Zhengji Li, and Zicheng Ma. 2022. Senti-exlm: Uyghur enhanced sentiment analysis model based on xlm. Electronics Letters, 58(13):517–519. find
  15. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… find
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  17. Antonio Martínez-García, Toni Badia, and Jeremy Barnes. 2021. Evaluating morphological typology in zero-shot cross-lingual transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International George A Miller. 1995. Wordnet: a lexical databas… find
  18. Oyesh Mann Singh, Sandesh Timilsina, Bal Krishna Bal, and Anupam Joshi. 2020. Aspect based abusive sentiment detection in nepali social media texts. In 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 301–308. find
  19. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lingu…
  20. Push Singh, Thomas Lin, Erik T Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. 2002. Open mind common sense: Knowledge acquisition from the general public. In On the Move to Meaningful Internet Systems 2002: CoopIS, DOA, and ODBASE: Confederated International Conferences CoopIS, DOA, and ODBASE… find
  21. Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3136–3153. find
  22. Hiroki Nakayama. 2018. seqeval: A python framework for sequence labeling evaluation. Software available from https://github.com/chakki-works/seqeval. ↗
  23. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. A…
  24. Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human…
  25. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487–503…
  26. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Li…
  27. Ayu Purwarianti and Ida Ayu Putu Ari Crisdayanti. 2019. Improving bi-lstm performance for indonesian sentiment analysis using paragraph vector. In 2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA), pages 1–5. IEEE. find
  28. Surangika Ranathunga and Isuru Udara Liyanage. 2021. Sentiment analysis of sinhala news comments. Transactions on Asian and Low-Resource Language Information Processing, 20(4):1–23. find
  29. Marina Sokolova, Nathalie Japkowicz, and Stan Szpakowicz. 2006. Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation. In Australasian joint conference on artificial intelligence, pages 1015–1021. Springer. find
  30. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
  32. Giorgos Vernikos and Andrei Popescu-Belis. 2021. Subword mapping and anchoring across languages. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2633–2647, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  33. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.
  34. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405–1418, Online. Associ…
  35. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  36. Wilson Wongso, David Samuel Setiawan, and Derwin Suhartono. 2021. Causal and masked language modeling of javanese language using transformer-based architectures. In 2021 International Conference on Advanced Computer Science and Information Systems (ICACSIS), pages 1–7. IEEE. find
  37. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics.
  38. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
  39. Yulei Zhu, Baima Luosai, Liyuan Zhou, Nuo Qun, and Tashi Nyima. 2023. Research on sentiment analysis of tibetan short text based on dual-channel hybrid neural network. In 2023 IEEE 4th International Conference on Pattern Recognition and Machine Learning (PRML), pages 377–384. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gurgurov-2024-adapting-multilingual-llms-low-resource/