Elamur Research⌕
☰
home / papers / 2022 / Siamese AraBERT-LSTM Model based Approach for Arabic Paraphrase Detection

Siamese AraBERT-LSTM Model based Approach for Arabic Paraphrase Detection

Summary

The paper addresses Arabic paraphrase detection, which means judging whether two sentences express the same meaning despite different wording. Arabic makes this harder because of its rich derivational morphology, missing vowels, homographs and agglutination, and because few public paraphrase corpora exist. The authors propose a Siamese setup in which both sentences pass through identical, weight-sharing branches. AraBERT supplies contextual subword embeddings, which are pooled and then modelled by an LSTM. The cosine similarity of the two resulting sentence vectors is mapped to a probability and compared against a threshold. To get training and test data, they build their own corpus from OSAC by replacing words with near neighbours from a word2vec-based vocabulary, constrained to the same part of speech. They also test on an Arabic SemEval set. AraBERT-LSTM reaches F1 0.8975 on the OSAC-derived corpus and 0.8650 on SemEval, beating GloVe-based and cosine-only variants. The work suggests that contextual subword embeddings combined with a small trained comparison network suit morphologically rich, low-resource languages.

Key points

  • Pipeline: AraBERT embeddings → mean pooling → shared LSTM → cosine similarity → probability compared with a threshold. Both sentences go through the same Siamese branch.
  • Corpus: passages from the Open Source Arabic Corpora (OSAC) are rewritten by swapping words for similar same-POS words, using word2vec plus POS weighting. The resulting pairs include obfuscation types such as same polarity and word addition or deletion.
  • F1 on OSAC: GloVe-Cosine 0.7750, AraBERT-Cosine 0.8050, GloVe-LSTM 0.8731, AraBERT-LSTM 0.8975.
  • F1 on SemEval: 0.7650, 0.7850, 0.8397 and 0.8650 for the same four models. The authors attribute the lower SemEval scores to its smaller vocabulary and fewer training examples.
  • Configuration:
    • AraBERT: 4 hidden layers, 4 attention heads, hidden size 256, dropout 0.1, Adam, 50 epochs, ReLU.
    • LSTM: 256 units, dropout 0.2, batch size 100.
    • The comparison table lists English systems (e.g., Meshram and Kumar BERT-BERT 0.896) measured on different datasets.

Interesting ideas

  • I1. A 2×2 ablation crosses the embedding type (static GloVe vs contextual AraBERT) with the comparison head (bare cosine vs trained LSTM), so each component's contribution is measured separately rather than only reporting the full system. — For Mova (learn): Evaluate each new parser or frame component as a factorial grid against a "dumb" baseline, so improvements can be attributed to a specific part instead of to the whole bundle.
  • I2. Paraphrase pairs are synthesised by replacing words only with embedding neighbours that share the same part of speech, which keeps sentence syntax intact while varying the lexicon. — For Mova (learn): POS-constrained (or UD-feature-constrained) lexical substitution could generate silver sentence pairs whose trees should stay structurally identical, serving as a consistency check for parsers.
  • I3. Scores on the self-generated corpus are noticeably higher than on the external SemEval set, a reminder that synthetic benchmarks can flatter a model. — For Mova (answer): Always pair self-built test sets for readers and QA with at least one independently authored set, and track the gap between them as a warning signal.
  • I4. The Siamese design uses one weight-sharing encoder for both inputs, so the comparison is symmetric and sentences are mapped into a common space before a simple distance is applied. — For Mova (frames): A shared encoder over two frame structures, followed by an inspectable similarity, could flag when two clauses describe the same event, which would assist coreference of events and paraphrase-aware frame merging.
  • I5. Subword-based contextual embeddings handle unseen morphological variants that word-level vectors miss, which the authors cite as the reason AraBERT beat GloVe on derivational Arabic. — For Mova (tokens): Back up the lemma lookup with subword evidence for out-of-vocabulary forms (rare derivations, neologisms), so the tagger degrades gracefully instead of failing on unknown words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Part-of-speech tagging

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mahmoud2022siamese,
  title = {{Siamese AraBERT-LSTM Model based Approach for Arabic Paraphrase Detection}},
  author = {Mahmoud, Adnen and Zrigui, Mounir},
  booktitle = {Proceedings of the 36th Pacific Asia Conference on Language, Information and Computation},
  year = {2022},
  url = {https://aclanthology.org/2022.paclic-1.60/}
}

References

36 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akbik A., Bergmann T., Blythe D., Rasul K., Schweter S., and Vollgraf R. 2019. An easy-to-use framework for state-of-the-art NLP, Conference of the North American Chapter of the Association for Computational Linguistics (demonstrations). find
  2. Aliane A. A., and Aliane H. 2020. Evaluating SIAMESE architecture neural models for Arabic textual similarity and plagiarism detection, 4 th International Symposium on Informatics and its Applications (ISIA), M’sila, Algeria: 1-8. find
  3. Ayata D., Saraclar M., and Ozgur A. 2017. BUSEM at SemEval-2017 Task 4 sentiment analysis with word embedding and long short term memory RNN approaches, 11 th International Workshop on Semantic Evaluations (SemEval-2017), Vancouver, Canada: 777–783. find
  4. Babi K., Martincic-Ipsi S., and Meštrovic A. 2020. Survey of neural text representation models, Information, volume.11: 1-32. find
  5. Gharavi E., Bijari K., Zahirnia K., and Veisi H. 2016. A deep learning approach to Persian plagiarism detection, FIRE (Working Notes): 1-6. find
  6. Haffar N., Hkiri E., and Zrigui M. 2020. Enrichment of Arabic TimeML corpus, International Conference on Computational Collective Intelligence (ICCCI), Da Nang, Vietnam: 655–667. find
  7. Haffar N., Ayadi R., Hkiri E., and Zrigui M. 2021. Temporal ordering of events via deep neural networks, 16 th International Conference on Document Analysis and Recognition (ICDAR), Lausanne, Switzerland: 762-777. find
  8. Hambi M., and Benabbou F. 2019. A new online plagiarism detection system based on deep learning, International Journal of Advanced Computer Science and Applications (IJACSA), 11(9): 470-478. find
  9. Hamza A., En-Nahnahi N., and Ouatik S. E. 2020. Contextual word representation and deep neural networks-based method for Arabic question classification, Advances in Science, Technology and Engineering Systems Journal, 5(5): 478-484. find
  10. He H., Gimpel K., and Lin J. 2015. Multi-perspective sentence similarity modeling with convolutional neural networks, Conference on Empirical Methods in Natural Language Processing (EMNLP): 1576–1586. find
  11. Hindocha E., Yazhiny V., Arunkumar A., and Boobalan P. 2019. Short-text semantic similarity using GloVe word embedding, International Research Journal of Engineering and Technology (IRJET), volume. 6: 553-558. find
  12. Iqbal A., Sharif O., Hoque M. M., and Sarker I. H. 2021. Word embedding based textual semantic similarity measure in Bengali, Procedia Computer Science, volume.193: 92–101. find
  13. Karaoglan D., Kisla T., and Metin S. K. 2016. Description of Turkish paraphrase corpus structure and generation method, International Conference on Computational Linguistics and Intelligent Text Processing (CICLing), Turkey: 208-217. find
  14. Liu L., Yang W., Rao J., Tang R., and Lin J. 2019. Incorporating contextual and syntactic structures improves semantic similarity modeling, Conference on Empirical Methods in Natural Language Processing and the 9 th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): 1204–1209. find
  15. Mahmoud A., and Zrigui M. 2017. Semantic similarity analysis for paraphrase identification in Arabic texts, 31 st Pacific Asia Conference on Language, Information and Computation (PACLIC), Philippine: 274-281 find
  16. Mahmoud A., and Zrigui M. 2019a. Sentence embedding and convolutional neural network for semantic textual similarity detection in Arabic language, Arabian for Engineering and Science Journal, volume. 44: 9263-9274. find
  17. Mahmoud A., and Zrigui M. 2019b. Similar meaning analysis for original documents identification in Arabic language, International Conference on Computational Collective Intelligence (ICCCI), Hendaye, France: 193–206. find
  18. Mahmoud A., and Zrigui M. 2021a. Arabic semantic textual similarity identification based on convolutional gated recurrent units, International Symposium on INnovations in Intelligent Systems and Applications (INISTA), Kocaeli, Turkey: 1-7. find
  19. Mahmoud A., and Zrigui M. 2021b. Hybrid attention-based approach for Arabic paraphrase detection, Applied Artificial Intelligence: 1-16. find
  20. Mahmoud A., and Zrigui M. 2021c. Semantic similarity analysis for corpus development and paraphrase detection in Arabic, International Arab Journal of Information Technology (IAJIT), volume. 18: 1-7. find
  21. Mahmoud A., and Zrigui M. 2021d. BLSTM-API: Bi-LSTM recurrent neural-based approach for Arabic paraphrase identification, Arabian for Science and Engineering, volume. 46: 4163-4174. find
  22. Meddeb O., Maraoui M., and Zrigui M. 2021. Arabic text documents recommendation using joint deep representations learning, Procedia Computer Science, 192(1): 812-821. find
  23. Meshram S., and Kumar M. A. 2021. Long short-term memory network for learning sentences similarity using deep contextual embeddings, International Journal of Information Technology, 13(4): 1633– 1641. find
  24. Mikolov T., Sutskever I., Corrado G., and Dean J. 2014. Distributed representations of words and phrases and their compositionality, arXiv:1405.4053. ↗ our card
  25. Mohammed S. M., Jacksi K., and Zeebaree S. R. M. 2019. A state-of-the-art survey on semantic similarity for document clustering using GloVe and density-based algorithms, Indonesian Journal of Electrical Engineering and Computer Science, 22(1): 552-562. find
  26. Nagoudi E. B. A., Khorsi H., Cherroun H., and Schwab D. 2018. A two-level plagiarism detection system for Arabic documents, Cybernetics and Information Technologies, In press, 18(1): 1-17. find
  27. Othman N., Faiz R., and Smaïli K. 2021. Learning English and Arabic question similarity with Siamese neural networks in community question answering services, Data and Knowledge Engineering (In press): 1-26. find
  28. Peinelt N., Nguyen D., and Liakata M. 2020. tBERT: Topic models and BERT joining forces for semantic similarity detection, 58 th Annual Meeting of the Association for Computational Linguistics: 7047–7055. find
  29. Pennington J., Socher R., and Manning C. 2014. GloVE: Global vectors for word representation, Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar: 1532-1543. find
  30. Peters M. E., Neumann M., Iyyer M., Gardner M., Clark C., Lee K., and Zettlemoyer L. 2018. Deep contextualized word representations, Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana: 2227-2237. find
  31. Ranashinghe T., Orasan O., and Mitkov R. 2019. Enhancing unsupervised sentence similarity methods with deep contextualized word representations, International Conference on Recent Advances in Natural Language Procesing (RANLP), Varna, Bulgaria: 994-1003. find
  32. Sghaier M. A., and Zrigui M. 2020. Rule-based machine translation from Tunisian dialect to modern Arabic standard, Procedia Computer Science, volume. 196: 310-319. find
  33. Shao Y. 2017. Hcti at semeval-2017 task 1: use convolutional neural network to evaluate semantic textual similarity, 11 th International Workshop on Semantic Evaluation (SemEval-2017): 130– 133. find
  34. Tai K. S., Socher R., and Manning C. D. 2015. Improved semantic representations from tree-structured long short-term memory networks, arXiv preprint arXiv:1503.00075. ↗
  35. Veisi H., Golchinpour M., Salehi M., and Gharavi E. 2022. Multi-level text document similarity estimation and its application for plagiarism detection, Iran Journal of Computer Science: 1-13. find
  36. Yalcin K., Cicekli I., and Ercan G. 2022. An external plagiarism detection system based on part-of-speech (POS) tag n-grams and word embedding, Expert Systems with Applications, volume. 197. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mahmoud-2022-siamese-arabert-lstm-model-based/