Elamur Research⌕
☰
home / papers / 2024 / WarwickNLP at SemEval-2024 Task 1: Low-Rank Cross-Encoders for Efficient Semantic Textual Relatedness

WarwickNLP at SemEval-2024 Task 1: Low-Rank Cross-Encoders for Efficient Semantic Textual Relatedness

Paper · 2024 · Fahad Ebrahim and Mike Joy

Summary

The paper describes the WarwickNLP entry for SemEval-2024 Task 1, Track A, a supervised regression task. Systems must predict a 0–1 score for how related two sentences are, in English and Moroccan Arabic. Instead of fully fine-tuning a large model, the authors keep a pre-trained cross-encoder frozen and train only a LoRA adapter (rank 8) on the task data. The base model is a RoBERTa cross-encoder that had already been trained on STS-B. After training, they merge the adapter into the model and use it to score test pairs.

They chose this setup through step-by-step development experiments:

  • first comparing several base models,
  • then comparing three adapter types,
  • then training for more epochs.

The final system ranked 8th in English (0.8425) and 7th in Moroccan Arabic (0.8163), beating the LabSE cosine-similarity baselines. This matters because a small trainable add-on to a model already suited to the task gives competitive results cheaply. That is especially useful when training data is scarce, as with Moroccan Arabic.

Key points

  • Data sizes: English has 5500 train / 250 dev / 2500 test pairs. Moroccan Arabic has only 925 / 70 / 427. This scarcity motivated the adapter, few-shot-style approach.
  • Base models, all with a Pfeiffer adapter (dev scores):
    • BERT 0.8023
    • ALBERT 0.7755
    • DeBERTa small 0.8088
    • T5 small 0.8003
    • RoBERTa base 0.8155
    • RoBERTa STSB-CE (cross-encoder trained on STS-B) 0.8296
  • Adapters on RoBERTa STSB-CE (dev): Pfeiffer 0.8296, Houlsby 0.8309, LoRA 0.8343. Raising training from 10 to 30 epochs lifted LoRA to 0.8417.
  • Official test scores: English 0.8425 vs. baseline 0.83; Moroccan Arabic 0.8163 vs. baseline 0.77. The baseline is LabSE embeddings compared by cosine similarity. Note that the paper calls its metric Spearman in some places and Pearson in others.
  • Side results:
    • CAMeLBERT, an Arabic pre-trained model, reached 0.871 on the Moroccan dev set and 0.8347 on test in post-evaluation, but was not submitted.
    • The same system applied to Algerian Arabic scored only 0.54.

Interesting ideas

  • I1. Starting from a model already trained on a neighbouring task (STS similarity) beat every generic pre-trained model, so the starting checkpoint can matter more than the backbone family.
  • I2. A low-rank adapter trains only small matrices next to frozen weights, producing a few-megabyte add-on per task that can be merged in or swapped out.
  • I3. Development proceeded as a controlled sequence: vary the base model, then the adapter type, then the epoch count, changing one factor at a time and keeping the best.
  • I4. Running the system on an unseen dialect (Algerian Arabic, 0.54) exposed how much performance depended on data overlap with training resources.
  • I5. A cross-encoder reads both sentences together with attention across them, which outperformed comparing two independently computed sentence embeddings.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): when gparse is adapted to a new UD dialect or genre, initialise from a parser trained on the closest existing treebank rather than from a raw encoder, and measure the gain.
  • I2 (tree): keep one frozen gparse backbone with small LoRA adapters per text domain (fables, school tests, word problems), and log which adapter produced each tree to keep the output explainable.
  • I3 (learn): structure training experiments as one-factor ablations with recorded dev scores, so each improvement can be attributed to a single change.
  • I4 (learn): add held-out, out-of-distribution text sets as negative controls, so silver-data or self-play gains are not mistaken for real generalisation.
  • I5 (answer): match a question against the text's frame/world structures jointly, aligning roles across both, instead of comparing separate summaries of each.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WarwickCoventry · United Kingdom8 works · 1,291 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksEfficient models & compression · 297 works

Cite

Cite as Fahad Ebrahim and Mike Joy. 2024. WarwickNLP at SemEval-2024 Task 1: Low-Rank Cross-Encoders for Efficient Semantic Textual Relatedness. Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.semeval-1.38

@inproceedings{ebrahim2024warwicknlp,
  title = {{WarwickNLP at SemEval-2024 Task 1: Low-Rank Cross-Encoders for Efficient Semantic Textual Relatedness}},
  author = {Fahad Ebrahim and Mike Joy},
  booktitle = {Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)},
  year = {2024},
  doi = {10.18653/v1/2024.semeval-1.38},
  url = {https://doi.org/10.18653/v1/2024.semeval-1.38}
}

References

20 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohamed Abdalla, Krishnapriya Vishnubhotla, and Saif Mohammad. 2023. What makes sentences semantically related? a textual relatedness dataset and empirical study. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 782–796. find
  2. Shima Asaadi, Saif Mohammad, and Svetlana Kiritchenko. 2019. Big bird: A large, fine-grained, bigram relatedness dataset for examining semantic composition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 505–516. find
  3. Tilman Beck, Bela Bohlender, Christina Viehmann, Vincent Hane, Yanik Adamson, Jaber Khuri, Jonas Brossmann, Jonas Pfeiffer, and Iryna Gurevych. 2022. AdapterHub playground: Simple and flexible few-shot learning with adapters. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 61–75, Dublin, Ireland. Association for Computational Linguistics. ↗
  4. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics. ↗
  5. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  7. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891.
  8. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654. ↗
  9. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. ↗
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. ↗
  11. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  12. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  14. Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Abinew Ali Ayele, Pavan Baswani, Meriem Beloucif, Chris Biemann, Sofia Bourhim, Christine De Kock, Genet Shanko Dekebo, Oumaima Hourrane, Gopichand Kanumolu, Lokesh Madasu, Samuel Rutunda, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Hailegnaw Getaneh Tilaye, Krishnapriya Vishnubhotla, Genta Winata, Seid Muhie Yimam, and Saif M. Mohammad. 2024a. Semrel2024: A collection of semantic textual relatedness datasets for 14 languages. find
  15. Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Meriem Beloucif, Christine De Kock, Oumaima Hourrane, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Krishnapriya Vishnubhotla, Seid Muhie Yimam, and Saif M. Mohammad. 2024b. SemEval-2024 task 1: Semantic textual relatedness for african and asian languages. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024). Association for Computational Linguistics. ↗
  16. Adrien Pavao, Isabelle Guyon, Anne-Catherine Letournel, Dinh-Tuan Tran, Xavier Baro, Hugo Jair Escalante, Sergio Escalera, Tyler Thomas, and Zhen Xu. 2023. Codalab competitions: An open source platform to organize scientific challenges. Journal of Machine Learning Research, 24(198):1–6. find
  17. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487–503, Online. Association for Computational Linguistics. ↗
  18. Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. 2023. Adapters: A unified library for parameter-efficient and modular transfer learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 149–160, Singapore. Association for Computational Linguistics. ↗
  19. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. ↗
  20. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. B The hyper-parameters used to develop the system are available in Table 7. The number of epochs during the development phase was 10, while in testing, it was increased to 30. Parameter Epoch Learning rate Batch size LoRA (r and alpha) GPU

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ebrahim-2024-warwicknlp-semeval-2024-task-1/