Elamur Research⌕
☰
home / papers / 2024 / Enhancing Semantic Similarity Understanding in Arabic NLP with Nested Embedding Learning

Enhancing Semantic Similarity Understanding in Arabic NLP with Nested Embedding Learning

Summary

Arabic had no sentence-embedding models trained with Matryoshka Representation Learning (MRL). MRL orders information inside a vector so that its leading dimensions still work on their own after the vector is cut short. The authors close this gap in two steps.

First, they used neural machine translation to render SNLI and MultiNLI into Arabic. They split the result into several subsets, among them anchor–positive pairs and anchor–positive–negative triplets, and they also translated sentence-similarity data for testing.

Second, they fine-tuned multilingual, Arabic-specific and English encoders on the Arabic NLI triplets with a nested objective, which yields Matryoshka versions of each model. They scored these on an Arabic STS benchmark using Pearson and Spearman correlations over cosine, Manhattan, Euclidean and dot-product similarity, at several truncation sizes.

The nested Arabic models beat the untuned originals by about 20–25%, and paraphrase-multilingual-mpnet-base-v2 was the strongest overall. The work gives Arabic NLP compact, size-adjustable embeddings, with the datasets and models released openly on Hugging Face.

Key points

  • The Arabic NLI resources come from NMT-translated SNLI and MultiNLI:
    • Pair subset: 314K train / 6.81K validation / 6.83K test.
    • Triplet subset: 558K train / 6.58K validation / 6.61K test.
  • The base encoders are paraphrase-multilingual-mpnet-base-v2, LaBSE, mpnet-base-all-nli-triplet, bert-base-arabertv02 and MARBERT. Each was turned into a Matryoshka variant, for example Arabic-labse-Matryoshka and Marbert-all-nli-triplet-Matryoshka.
  • Scoring used the EmbeddingSimilarityEvaluator with Pearson and Spearman correlations under four similarity functions. Embedding sizes ranged over 768, 512, 256, 128 and 64.
  • Ranking of the models:
    • paraphrase-multilingual-mpnet-base-v2 led on most dimensions and metrics.
    • LaBSE was stable, especially at larger sizes.
    • bert-base-arabertv02 and MARBERT scored lowest.
  • Error analysis showed that unrelated sentence pairs get inflated similarity scores, i.e. false positives. A Gradio demo on Hugging Face lets users choose the model, the comparison mode and the vector size.

Interesting ideas

  • I1. Train one vector whose short prefix is already meaningful, then use the cheap prefix to shortlist candidates and the full vector to rerank the survivors; this trades cost against accuracy without keeping separate models. — For Mova (world): when linking a mention to WordNet/Wikidata entries, a short-prefix lookup could propose a few candidates, and the explainable symbolic checks would still make the final decision.
  • I2. Break evaluation down by gold similarity band (none, low, moderate) instead of reporting a single correlation; this exposed a bias toward overrating unrelated pairs that an average score would have hidden. — For Mova (learn): report accuracy per difficulty or category bucket and always include a band of clearly unrelated inputs as a negative control, so over-matching shows up directly.
  • I3. Score the same model under several similarity functions (cosine, Manhattan, Euclidean, dot product) to check that conclusions do not depend on one arbitrary metric choice. — For Mova (learn): when comparing parser or frame outputs, compute agreement with more than one measure so a model's ranking is not an artifact of a single scoring rule.
  • I4. Turn NLI data into triplets in which the negative is a sentence that is topically close but contradicts the anchor; such near-miss negatives teach fine distinctions better than random ones. — For Mova (frames): contradiction triplets from SNLI/MultiNLI could serve as test cases checking that the frames for anchor and negative really differ in who did what, not only in wording.
  • I5. Bootstrap training material for a low-resource setting by machine-translating an established English corpus while keeping its labels. — For Mova (back): the original English NLI pairs and their translations give aligned material for checking whether text → graph → text round-trips preserve the labelled relation (entailment or contradiction).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

7 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.21139 · arXiv 2407.21139

@misc{nacar2024enhancing,
  title = {{Enhancing Semantic Similarity Understanding in Arabic NLP with Nested Embedding Learning}},
  author = {Nacar, Omer and Koubaa, Anis},
  year = {2024},
  doi = {10.48550/arxiv.2407.21139},
  eprint = {2407.21139},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.21139}
}

References

23 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521(7553):436–444, 2015. find
  2. P. Nayak. Understanding searches better than ever before. Google AI Blog, 2019. URL https://blog.google/products/search/search-language-understanding-bert/. ↗
  3. J. Dean. Challenges in building large-scale information retrieval systems. In Keynote of the 2nd ACM International Conference on Web Search and Data Mining (WSDM), volume 10, 2009. find
  4. C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017. find
  5. Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., ... & Farhadi, A. (2022). Matryoshka representation learning. Advances in Neural Information Processing Systems, 35, 30233-30249. find
  6. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3980–3990. Association for Computational Linguistics. https://huggingface.co/spaces/Omartificial-Intelligence-Space/Arabic-Sentence-Similarity-Matryoshka-Models ↗
  7. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. find
  8. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778, 2016. find
  9. M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297–304. JMLR Workshop and Conference Proceedings, 2010. find
  10. M. G. Harris and C. D. Giachritsis. Coarse-grained information dominates fine-grained information in judgments of time-to-contact from retinal flow. Vision research, 40(6):601–611, 2000. find
  11. C. Waldburger. As search needs evolve, microsoft makes ai tools for better search available to researchers and developers. Microsoft AI Blog, 2019. URL https://blogs.microsoft.com/ai/bing-vector-search/. ↗
  12. Y. A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4):824–836, 2018. find our card
  13. H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791, 2019. ↗
  14. M. Wallingford, H. Li, A. Achille, A. Ravichandran, C. Fowlkes, R. Bhotika, and S. Soatto. Task adaptive parameter sharing for multi-task learning. arXiv preprint arXiv:2203.16708, 2022. ↗
  15. J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang. Slimmable neural networks. arXiv preprint arXiv:1812.08928, 2018. ↗
  16. O. Rippel, M. Gelbart, and R. Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pages 1746–1754. PMLR, 2014. find
  17. Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015). A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. ↗
  18. Kim, S., Kang, I., & Kwak, N. (2019, July). Semantic sentence matching with densely-connected recurrent and co-attentive information. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 6586-6593). find
  19. MacCartney, B., & Manning, C. D. (2008, August). Modeling semantic containment and exclusion in natural language inference. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008) (pp. 521-528). find
  20. Yang, Y., Yuan, S., Cer, D., Kong, S. Y., Constant, N., Pilar, P., ... & Kurzweil, R. (2018). Learning semantic textual similarity from conversations. arXiv preprint arXiv:1804.07754. ↗
  21. Kudo, T., & Richardson, J. (2018). Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. ↗ our card
  22. Klein, G., Kim, Y., Deng, Y., Senellart, J., & Rush, A. M. (2017). Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810. ↗ our card
  23. Henderson, M., Al-Rfou, R., Strope, B., Sung, Y. H., Lukács, L., Guo, R., ... & Kurzweil, R. (2017). Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nacar-2024-enhancing-semantic-similarity-understanding-arabic/