Elamur Research⌕
☰
home / papers / 2022 / Data Augmentation using Transformers and Similarity Measures for Improving Arabic Text Classification

Data Augmentation using Transformers and Similarity Measures for Improving Arabic Text Classification

Summary

Arabic text classifiers often lack enough labelled data, and their sentiment datasets tend to be skewed toward some classes. Earlier Arabic data augmentation (DA) work mostly relied on rule-based paraphrasing or on noising such as word insertion and replacement. These methods can damage grammar or meaning in a morphologically rich language. The authors instead use the generative transformer AraGPT-2 to produce new training sentences from existing ones. Each candidate is scored against its source with four similarity measures: Euclidean, cosine, Jaccard and BLEU. These cover context, semantics, diversity and novelty, and only candidates within the chosen thresholds are kept. Every resulting augmented dataset is then used to fine-tune AraBERT for sentiment classification. Experiments on four Arabic corpora of different size, label count and imbalance show F1 gains on all of them, from about 4 to 13 points. The work matters because it shows that pretrained generators plus similarity-based filtering can grow scarce, unbalanced non-English data without hand-built resources such as synonym dictionaries.

Key points

  • Pipeline: AraGPT-2 generates paraphrase candidates, four similarity measures (Euclidean, cosine, Jaccard, BLEU) filter them, and AraBERT is fine-tuned and evaluated on each augmented set. AraBERT also handles preprocessing.
  • Datasets: AraSarcasm, ASTD, ATT and MOVIE. They differ in size, number of labels and degree of class imbalance.
  • F1 improvements over the non-augmented baseline: about 4% on AraSarcasm, 6% on ASTD, 9% on ATT and 13% on MOVIE. The method also beat a prior Arabic DA study evaluated on the same data.
  • Which filter worked best depended on dataset size. BLEU-filtered augmentation gave the best classification on large datasets, and Jaccard-filtered augmentation gave the best on small ones.
  • Accepted thresholds came out higher for cosine than for the other three measures. The authors attribute this to cosine being largely insensitive to sentence length. Larger growth of the augmented set generally went together with larger classification gains.

Interesting ideas

  • I1. Generate many candidates with a language model, then accept only those whose similarity to the source stays within a threshold; this keeps the output close in meaning but not a near-copy, which is a general recipe for synthetic data. — For Mova (learn): Silver sentences produced by an LLM teacher could pass a similarity gate against their seed sentence before the witness parsers vote on them, so meaning-drifting rewrites are dropped cheaply and early.
  • I2. Use several similarity measures at once, because each captures a different quality: token overlap (Jaccard), n-gram fidelity (BLEU), vector-space meaning (cosine) and raw distance (Euclidean). Each can then be read as a separate, explainable reason for accepting or rejecting a sentence. — For Mova (back): The text → graph → text compressor could report BLEU/Jaccard for surface faithfulness and an embedding cosine for meaning separately, making it visible whether a round-trip failure is lexical or semantic.
  • I3. Metrics that ignore sentence length (cosine) accept more candidates at the same nominal strictness than length-sensitive ones. Thresholds should therefore be calibrated per metric rather than shared. — For Mova (learn): Any similarity-based filter or evaluation in training should get per-metric, possibly length-bucketed thresholds, so long sentences are not systematically favoured or penalised.
  • I4. The best filtering criterion depended on corpus size: a looser overlap measure suited small data and a stricter n-gram measure suited large data. Augmentation policy should be tuned to the data regime rather than fixed. — For Mova (learn): When building silver sets for rare constructions with few seeds versus common ones with many, Mova could switch acceptance criteria and check the effect against negative controls.
  • I5. Targeting augmentation at under-represented classes counters label imbalance without collecting new annotations. — For Mova (frames): Rare pragmatic categories such as irony or uncommon speech acts could be expanded with filtered paraphrases, so the prag classifiers see more balanced examples while each added sentence stays traceable to its source.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2212.13939 · arXiv 2212.13939

@misc{refai2022data,
  title = {{Data Augmentation using Transformers and Similarity Measures for Improving Arabic Text Classification}},
  author = {Refai, Dania and Abo-Soud, Saleh and Abdel-Rahman, Mohammad},
  year = {2022},
  doi = {10.48550/arxiv.2212.13939},
  eprint = {2212.13939},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.13939}
}

References

61 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. N. Ranjan, K. Mundada, K. Phaltane, and R. Ahmad, “A survey on techniques in NLP,” International Journal of Computer Applications, vol. 134, no. 8, pp. 6–9, January 2016. find
  2. U. Kamath, J. Liu, and J. Whitaker, Deep learning for NLP and speech recognition. Springer, June 2019, vol. 84. find
  3. R. Socher, Y. Bengio, and D. Manning, Christopher, “Deep learning for NLP (without magic),” Tutorial Abstracts of ACL, p. 5, July 2012. find
  4. J. Wei and K. Zou, “EDA: Easy data augmentation techniques for boosting performance on text classification tasks,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), November 2019, pp. 6382–6388. find our card
  5. G. Daval-Frerot and Y. Weis, “WMD at SemEval-2020 tasks 7 and 11: Assessing humor and propaganda using unsupervised data augmentation,” in Proceedings of the Fourteenth Workshop on Semantic Evaluation, December 2020, pp. 1865–1874. find
  6. X. Dai and H. Adel, “An analysis of simple data augmentation for named entity recognition,” in Proceedings of the International Conference on Computational Linguistics, December 2020, pp. 3861–3867. find our card
  7. J.-P. Corbeil and A. Ghadivel, Hadi, “BET: A backtranslation approach for easy data augmentation in transformer-based paraphrase identification context,” in arXiv:2009.12452, September 2020. ↗
  8. C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, no. 1, pp. 1–48, December 2019. find
  9. S. Y. Feng, V. Gangal, J. Wei, S. Chandar, S. Vosoughi, T. Mitamura, and E. Hovy, “A survey of data augmentation approaches for NLP,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP, August 2021, pp. 968–988. find our card
  10. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems, vol. 30, 2017. find
  11. T. Lin, Y. Wang, X. Liu, and X. Qiu, “A survey of transformers,” AI Open, vol. 3, pp. 111–132, October 2022. find
  12. Y. Hou, Y. Liu, W. Che, and T. Liu, “Sequence-to-sequence data augmentation for dialogue language understanding,” in Proceedings of the International Conference on Computational Linguistics, August 2018, pp. 1234–1245. find
  13. Y. Hou, S. Chen, W. Che, C. Chen, and T. Liu, “C2C-genda: cluster-to-cluster generation for data augmentation of slot filling,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 14, May 2021, pp. 13 027–13 035. find
  14. K. Li, C. Chen, X. Quan, Q. Ling, and Y. Song, “Conditional augmentation for aspect term extraction via masked sequence-to-sequence generation,” in Proceedings of the Annual Meeting of the Association for Computational Linguistics, July 2020, pp. 7056–7066. find
  15. T. Kober, J. Weeds, L. Bertolini, and D. Weir, “Data augmentation for hypernymy detection,” in Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, April 2021, pp. 1034–1048. find
  16. A. Celikyilmaz, E. Clark, and J. Gao, “Evaluation of text generation: A survey,” arXiv:2006.14799 , June 2020. ↗
  17. P.-E. Danielsson, “Euclidean distance mapping,” Computer Graphics and Image Processing, vol. 14, no. 3, pp. 227–248, November 1980. find
  18. J. Zobel and A. Moffat, “Exploring the similarity space,” ACM SIGIR Forum, vol. 32, no. 1, pp. 18–34, April 1998. find
  19. G. I. Ivchenko and S. A. Honov, “On the Jaccard similarity test,” Journal of Mathematical Sciences, vol. 88, no. 6, pp. 789–794, March 1998. find
  20. R. S. Papineni, Kishore, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in Proceedings of the Annual Meeting on Association for Computational Linguistics, 2002, p. 311–318. find our card
  21. A. R. Fabbri, S. Han, H. Li, H. Li, M. Ghazvininejad, S. Joty, D. Radev, and Y. Mehdad, “Improving zero and few-shot abstractive summarization with intermediate fine-tuning and data augmentation,” in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2021, pp. 704–717. find
  22. C. Rastogi, N. Mofid, and F.-I. Hsiao, “Can we achieve more with less? exploring data augmentation for toxic comment classification,” arXiv:2007.00875, July 2020. ↗
  23. G. Yan, Y. Li, S. Zhang, and Z. Chen, “Data augmentation for deep learning of judgment documents,” in International Conference on Intelligent Science and Big Data Engineering, October 2019, pp. 232–242. find
  24. Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, “Unsupervised data augmentation for consistency training,” Advances in Neural Information Processing Systems, vol. 33, pp. 6256–6268, July 2020. find our card
  25. I. Al-Huri, “Arabic language: historic and sociolinguistic characteristics,” English Literature and Language Review, vol. 1, no. 2, pp. 28–36, 2015. find
  26. Ethnologue. (2020) Summary by language size. [Online]. Available: https://www.ethnologue.com/statistics/summary-language size- 19 ↗
  27. K. A. Wahdan, S. Hantoobi, S. A. Salloum, and K. Shaalan, “A systematic review of text classification research based on deep learning models in Arabic language,” International Journal of Electrical and Computer Engineering, vol. 10, no. 6, pp. 6629–6643, December 2020. find
  28. R. Duwairi and F. Abushaqra, “Syntactic-and morphology-based text augmentation framework for Arabic sentiment analysis,” PeerJ Computer Science, vol. 7, p. e469, April 2021. find
  29. E. Williams, P. Rodrigues, and S. Tran, “Accenture at CheckThat! 2021: interesting claim identification and ranking with contextually sensitive lexical training data augmentation,” arXiv:2107.05684, July 2021. ↗
  30. A. Israeli, Y. Nahum, S. Fine, and K. Bar, “The IDC system for sentiment classification and sarcasm detection in Arabic,” in Proceedings of the Arabic Natural Language Processing Workshop, April 2021, pp. 370–375. find our card
  31. C. Sabty, I. Omar, F. Wasfalla, M. Islam, and S. Abdennadher, “Data augmentation techniques on Arabic data for named entity recognition,” Procedia Computer Science, vol. 189, pp. 292–299, January 2021. find
  32. W. Antoun, F. Baly, and H. Hajj, “AraGPT2: Pre-trained transformer for Arabic language generation,” in Proceedings of the Arabic Natural Language Processing Workshop, April 2021, pp. 196–207. find
  33. W. Anton, F. Baly, and H. Hajj, “AraBERT: Transformer-based model for Arabic language understanding,” in Proceedings of the LREC Workshop Language Resources and Evaluation Conference, May 2020, p. 9. find our card
  34. K. Gaanoun and I. Benelallam, “Arabic dialect identification: An Arabic-BERT model with data augmentation and ensembling strategy,” in Proceedings of the Arabic Natural Language Processing Workshop, December 2020, pp. 275–281. find
  35. F. Harrag, M. Dabbah, K. Darwish, and A. Abdelali, “BERT transformer model for detecting Arabic GPT2 auto-generated tweets,” in Proceedings of the Arabic Natural Language Processing Workshop, December 2020, pp. 207–214. find
  36. A. Abuzayed and H. Al-Khalifa, “Sarcasm and sentiment detection in Arabic tweets using BERT-based models and data augmentation,” in Proceedings of the Arabic Natural Language Processing Workshop, April 2021, pp. 312–317. find our card
  37. W. Al-Jamal, A. M. Mustafa, and M. Z. Ali, “Sarcasm detection in arabic short text using deep learning,” in Proceedings of International Conference on Information and Communication Systems (ICICS), 2022, pp. 362–366. find
  38. S. AlAwawdeh and G. Abandah, “Improving the accuracy of semantic similarity prediction of arabic questions using data augmentation and ensemble,” in Proceedings IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), 2021, pp. 272–277. find
  39. W. Antoun, F. Baly, and H. Hajj, “AraELECTRA: Pre-training text discriminators for Arabic language understanding,” in Proceedings of the Arabic Natural Language Processing Workshop, April 2021, pp. 191–195. find
  40. X. Carrasco, A. Elnagar, and M. Lataifeh, “A generative adversarial network for data augmentation: The case of Arabic regional dialects,” Procedia Computer Science, vol. 189, pp. 92–99, January 2021. find
  41. K. Kowsari, K. J. Meimandi, M. Heidarysafa, S. Mendu, L. Barnes, and D. Brown, “Text classification algorithms: A survey,” Information, vol. 10, no. 4, p. 150, April 2019. find
  42. F. Abdul-Ghani and N. A. Abdullah, “A survey on Arabic text classification using deep and machine learning algorithms,” Iraqi Journal of Science, vol. 63, no. 1, pp. 409–419, January 2022. find
  43. M. Sayed, R. K. Salem, and A. E. Khder, “A survey of Arabic text classification approaches,” International Journal of Computer Applications in Technology, vol. 59, no. 3, pp. 236–251, 2019. find
  44. A. Abdelali, K. Darwish, N. Durrani, and H. Mubarak, “Farasa: A fast and furious segmenter for Arabic,” in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, June 2016, pp. 11–16. find our card
  45. M. Al-Yahya, H. Al-Khalifa, H. Al-Baity, D. AlSaeed, and A. Essam, “Arabic fake news detection: a comparative study of neural networks and transformer-based approaches,” Complexity, vol. 2021, January 2021. find
  46. M. Abdul-Mageed and A. Elmadany, “ARBERT & MARBERT: Deep bidirectional transformers for Arabic,” in Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, vol. 1, August 2021, pp. 7088–7105. find our card
  47. J. Chen, D. Tam, C. Raffel, M. Bansal, and D. Yang, “An empirical survey of data augmentation for limited data learning in NLP,” arXiv:2106.07499, 2021. ↗ our card
  48. B. Li, Y. Hou, and W. Che, “Data augmentation approaches in natural language processing: A survey,” AI Open, vol. 3, pp. 71–90, March 2022. find
  49. I. A. Farha, W. Zaghouani, and W. Magdy, “Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic,” in Proceedings of the Arabic Natural Language Processing Workshop, April 2021, pp. 296–305. find our card
  50. S. Malaysha, M. Jarrar, and M. Khalilia, “Context-gloss augmenta- tion for improving arabic target sense verification,” arXiv preprint arXiv:2302.03126, 2023. ↗
  51. H. ElSahar and R. El-Beltagy, Samhaa, “Building large Arabic multi-domain resources for sentiment analysis,” in International Conference on Intelligent Text Processing and Computational Linguistics, April 2015, pp. 23–34. find
  52. F. Almeida and G. Xexéo, “Word embeddings: A survey,” arXiv:1901.09069, January 2019. ↗
  53. H. Almerekhi and T. Elsayed, “Detecting automatically-generated Arabic tweets,” Information Retrieval Technology, pp. 123–134, December 2015. find
  54. N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using siamese BERT-networks,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), November 2019, pp. 3982–3992. find
  55. L. Zhang, S. Wang, and B. Liu, “Deep learning for sentiment analysis: A survey,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 8, no. 4, p. e1253, July 2018. find
  56. D. Anguita, L. Ghelardoni, A. Ghio, L. Oneto, and S. Ridella, “The ‘ k ’ in k -fold cross validation,” in Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), 2012, pp. 441–446. find
  57. M. Nabil, M. Aly, and A. Atiya, “ASTD: Arabic sentiment tweets dataset,” in Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing, September 2015, pp. 2515–2519. find our card
  58. A. Jindal, A. G. Chowdhury, A. Didolkar, D. Jin, R. Sawhney, and R. Shah, “Augmenting NLP models using latent feature interpolations,” in Pro- ceedings of the International Conference on Computational Linguistics, December 2020, pp. 6931–6936. find
  59. P. Liu, X. Wang, C. Xiang, and W. Meng, “A survey of text data augmentation,” in Proceedings of the International Conference on Computer Communication and Network Security (CCNS), August 2020, pp. 191– 195. find
  60. D. Refai, “Arabic-data-augmentation,” https://github.com/Dania-Refai/Arabic-Data-Augmentation.git, 2023, [Source code]. ↗
  61. A. Ross and V. L. Willson, Paired Samples T-Test. Rotterdam: SensePublishers, 2017, pp. 17–19. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/refai-2022-data-augmentation-using-transformers-similarity/