Elamur Research⌕
☰
home / papers / 2022 / Arabic Keyphrase Extraction: Enhancing Deep Learning Models with Pre-trained Contextual Embedding and External Features

Arabic Keyphrase Extraction: Enhancing Deep Learning Models with Pre-trained Contextual Embedding and External Features

Paper · 2022 · Randah Alharbi and Husni Al-Muhtasab

Summary

The paper addresses automatic keyphrase extraction for Arabic, an area that lags behind English in adopting modern neural methods. The authors cast the task as token-level sequence labeling: a Bi-LSTM tags each token as inside or outside a keyphrase. They compare two kinds of pre-trained token representations, static Word2Vec vectors and contextual BERT embeddings. BERT is used in three ways: the last encoder layer as features, the last four layers concatenated, and full fine-tuning. On top of the embeddings they add hand-made features. These are linguistic (POS), positional (first occurrence and sentence order) and statistical (TFIDF), tried alone and in combinations.

Experiments on the ArabicKPE dataset show that contextual embeddings bring a large gain. The best system, BERT with POS and positional features, reaches 0.45 keyphrase-level F1. Extra features help Word2Vec far more than BERT. The authors present this as the first use of BERT contextual embeddings for Arabic keyphrase extraction, giving a stronger baseline for indexing and summarization in Arabic.

Key points

  • Setup: Bi-LSTM sequence tagger, ArabicKPE dataset. Embeddings come from Word2Vec or BERT, and the added features are POS, first occurrence, sentence order and TFIDF.
  • Under the evaluation protocol of Helmy et al. (2018), which takes the top 15 predictions and stems with NLTK ISRIStemmer, keyphrase-level F1 is:
    • 0.26 for the earlier model;
    • 0.25 for Word2Vec without features and 0.33 for its best feature set;
    • 0.47 for BERT without features and 0.50 for its best feature set.
  • Keyword-level F1 for the best BERT model is 0.64.
  • Feature effects on BERT:
    • Keyphrase-level scores stay within 0.42–0.45 across all feature combinations.
    • In 6 of 11 combinations the score is unchanged from the no-feature model.
    • The authors suggest BERT may already encode much of this information.
  • TFIDF tends to hurt Word2Vec at the full-phrase level, even when it raises keyword-level scores.
  • Stated limitations:
    • The keyphrase metric is strict exact match.
    • Word-level POS values are copied onto BERT subword pieces.
    • GPU nondeterminism makes exact reproduction difficult.

Interesting ideas

  • I1. Score every model at two granularities, single keywords and whole exact-match phrases; a large gap between them shows the model finds the right words but gets boundaries or order wrong (larger for Word2Vec than for BERT).
  • I2. A systematic feature ablation showed that adding explicit features to a contextual encoder often changes nothing, which hints the encoder already carries that information.
  • I3. Word-level linguistic tags were copied onto each subword piece, and the authors checked that dropping this copying did not alter results.
  • I4. Two positional cues, first occurrence and sentence order, behaved differently depending on the encoder: BERT gained more from first occurrence and Word2Vec more from sentence order.
  • I5. To compare fairly with prior work, the authors reproduced the earlier paper's protocol (top-15 cutoff, stemming) rather than relying only on their own metric.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluate span-producing steps such as entity mentions or frame arguments both per token and per complete span, so that boundary errors are reported separately from membership errors.
  • I2 (tree): Before feeding en-tagger POS or morphology into gparse, run a similar ablation to check whether those inputs are redundant with the transformer and so not worth their complexity.
  • I3 (tree): When aligning word-level tags from crate en with gparse subword tokens, test both the broadcast and the no-broadcast variants instead of assuming one is correct.
  • I4 (world): Use first-mention position as a transparent salience signal when ranking which entities are central to the text's world, and compare it against sentence-order cues.
  • I5 (answer): When reporting scores on school tests or fable QA against published systems, rerun their exact scoring rules so that the comparisons stay honest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Fahd University of Petroleum and MineralsDhahran · Saudi Arabia52 works · 4,666 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Randah Alharbi and Husni Al-Muhtasab. 2022. Arabic Keyphrase Extraction: Enhancing Deep Learning Models with Pre-trained Contextual Embedding and External Features. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.30

@inproceedings{alharbi2022arabic,
  title = {{Arabic Keyphrase Extraction: Enhancing Deep Learning Models with Pre-trained Contextual Embedding and External Features}},
  author = {Randah Alharbi and Husni Al-Muhtasab},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.30},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.30}
}

References

56 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wael Al Etaiwi, Arafat A. Awajan, and Dima Suleiman. 2019. Keywords extraction from arabic documents using centrality measures. In 2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS), pages 237–241. find
  2. Meran M. Al Hadidi, Muath Alzghool, and Hasan Muaidi. 2019. Keyword extraction from arabic text using the page rank algorithm. International Journal of Innovative Technology and Exploring Engineering, 8(12):3495–3504. find
  3. Mohammed Al-Logmani and Husni Al-Muhtaseb. Arabic dataset for automatic keyphrase extraction. In Second International Conference on Software Engineering (SOEN-2017), pages 217–222. find
  4. Nidaa Ghalib Ali and Nazlia Omar. 2015. A hybrid of statistical and machine learning methods for Arabic keyphrase extraction. Asian Journal of Applied Sciences, 8(4):269–276. ↗
  5. Fahad Mazaed Alotaibi and Shakeel Ahmad. 2019. Keywords Extraction from the Text of Holy Quran Using Linguistic and Heuristic Rules. International Journal of Computer Science and Network Security, 19(2):82– 87. find
  6. Rabah Alzaidy, Cornelia Caragea, and C. Lee Giles. 2019. Bi-lstm-crf sequence labeling for keyphrase extraction from scholarly documents. In The World Wide Web Conference, WWW ’19, page 2551–2557, New York, NY, USA. Association for Computing Machinery. find
  7. Eslam Amer and Khaled Foad. 2017. Akea: An arabic keyphrase extraction algorithm. In Proceedings of the International Conference on Advanced Intelligent Systems and Informatics 2016, pages 137–146, Cham. Springer International Publishing. find
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  9. Batool Armouty and Sara Tedmori. 2019. Automated keyword extraction using support vector machine from arabic news documents. In 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), pages 342–346. find
  10. Arafat Awajan. 2015. Keyword extraction from arabic documents using term equivalence classes. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 14(2). ↗
  11. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. ↗
  12. Marco Basaldella, Elisa Antolli, Giuseppe Serra, and Carlo Tasso. 2018. Bidirectional lstm recurrent neural network for keyphrase extraction. In Digital Libraries and Multimedia Archives, pages 180–187, Cham. Springer International Publishing. find
  13. Marco Basaldella, Muhammad Helmy, Elisa Antolli, Mihai Horia Popescu, Giuseppe Serra, and Carlo Tasso. 2017. Exploiting and evaluating a supervised, multilanguage keyphrase extraction pipeline for under-resourced languages. International Conference Recent Advances in Natural Language Processing, RANLP, 2017-Septe(1998):78–85. find
  14. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620, Hong Kong, China. Association for Computational Linguistics. find
  15. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  16. Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Jatowt. 2020. Yake! keyword extraction from single documents using multiple local features. Information Sciences, 509:257–289. ↗
  17. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Association for Computational Linguistics. ↗
  18. Kareem Darwish and Walid Magdy. 2014. Arabic Information Retrieval. Foundations and Trends® in Information Retrieval, 7(4):239–342. ↗
  19. Cristian Dascalu and Ştefan Trăuşan-Matu. 2021. Experiments with contextualized word embeddings for keyphrase extraction. In 2021 23rd International Conference on Control Systems and Computer Science (CSCS), pages 447–452. find
  20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  21. Liangping Ding, Zhixiong Zhang, and Yang Zhao. 2021. Bert-based chinese medical keyphrase extraction model enhanced with external features. In Towards Open and Trustworthy Digital Societies: 23rd International Conference on Asia-Pacific Digital Libraries, ICADL 2021, Virtual Event, December 1–3, 2021, Proceedings, page 167–176, Berlin, Heidelberg. Springer-Verlag. find
  22. Samhaa R. El-Beltagy and Ahmed Rafea. 2009. Kp-miner: A keyphrase extraction system for english and arabic documents. Information Systems, 34(1):132– 144. ↗
  23. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631– 1640, Berlin, Germany. Association for Computational Linguistics.
  24. Nizar Y. Habash. 2010. Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  25. Dana Halabi and Arafat Awajan. 2019. Graph-Based Arabic Key-phrases Extraction. In 2019 2nd International Conference on new Trends in Computing Sciences (ICTCS), Amman, Jordan. IEEE. find
  26. Kazi Saidul Hasan and Vincent Ng. 2014. Automatic keyphrase extraction: A survey of the state of the art. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1262–1273, Baltimore, Maryland. Association for Computational Linguistics.
  27. Muhammad Helmy, Marco Basaldella, Eddy Maddalena, Stefano Mizzaro, and Gianluca Demartini. 2016. Towards building a standard dataset for arabic keyphrase extraction evaluation. In 2016 International Conference on Asian Language Processing (IALP), pages 26–29. find
  28. Muhammad Helmy, R.M. Vigneshram, Giuseppe Serra, and Carlo Tasso. 2018. Applying deep learning for arabic keyphrase extraction. Procedia Computer Science, 142:254–261. Arabic Computational Linguistics. find
  29. Wei Zhang Jiqing Yao Yuquan Le Kehua Yang, Yaodong Wang. 2019. Keyphrase generation based on self-attention mechanism. Computers, Materials & Continua, 61(2):569–581. ↗
  30. Yeonsoo Lim, Deokjin Seo, and Yuchul Jung. 2020. Fine-tuning bert models for keyphrase extraction in scientific articles. JOURNAL OF ADVANCED INFORMATION TECHNOLOGY AND CONVER-GENCE, 10:45–56. ↗
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. ↗
  32. Mourad Loukam, Djamila Hammouche, Freha Mez-zoudj, and Fatma Zohra Belkredim. 2019. Keyphrase extraction from modern standard arabic texts based on association rules. In Arabic Language Processing: From Theory to Practice, pages 209–220, Cham. Springer International Publishing. find
  33. Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 582–592, Vancouver, Canada. Association for Computational Linguistics.
  34. Zakariae Alami Merrouni, Bouchra Frikh, and Brahim Ouhbi. 2020. Automatic keyphrase extraction: a survey and trends. Journal of Intelligent Information Systems, 54(2):391–424. ↗
  35. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, page 3111–3119, Red Hook, NY, USA. Curran Associates Inc. ↗
  36. Funan Mu, Zhenting Yu, LiFeng Wang, Yequan Wang, Qingyu Yin, Yibo Sun, Liqun Liu, Teng Ma, Jing Tang, and Xing Zhou. 2020. Keyphrase extraction with span-based feature representations. ↗
  37. Dhiaa Musleh, Rashad Ahmed, Atta Rahman, and Fahd Al-Haidari. 2019. A novel approach to arabic keyphrase extraction. ICIC Express Letters, 10:875– 884. find
  38. Hassan Najadat, Ismail Hmeidi, Mohammed N. Al-Kabi, and Maysa Mahmoud Bany Issa. 2016. Automatic keyphrase extractor from arabic documents. International Journal of Advanced Computer Science and Applications, 7. ↗
  39. Narjes Nikzad-Khasmakhi, Mohammad-Reza Feizi-Derakhshi, Meysam Asgari-Chenaghlu, Mohammad-Ali Balafar, Ali-Reza Feizi-Derakhshi, Taymaz Rahkar-Farshi, Majid Ramezani, Zoleikha Jahanbakhsh-Nagadeh, Elnaz Zafarani-Moattar, and Mehrdad Ranjbar-Khadivi. 2021. Phraseformer: Multimodal key-phrase extraction using transformer and graph embedding. ↗
  40. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  41. Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised learning of sentence embeddings using compositional n-gram features. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 528–540, New Orleans, Louisiana. Association for Computational Linguistics.
  42. Eirini Papagiannopoulou and Grigorios Tsoumakas. 2020. A review of keyphrase extraction. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(2):1–45. ↗
  43. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  44. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. Mohammad Taher Pilehvar and Jose Camacho-Collados. Embeddings in Natural Language Processing: Theory and Advances in Vector Representations of Meaning. Springer Cham.
  45. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  46. Mahmoud Rammal, Zeinab Bahsoun, and Mona Jabbour. 2015. Keyword extraction from arabic legal texts. Interactive Technology and Smart Education, 12:62–71. ↗
  47. Dhruva Sahrawat, Debanjan Mahata, Haimin Zhang, Mayank Kulkarni, Agniv Sharma, Rakesh Gosangi, Amanda Stent, Yaman Kumar, Rajiv Ratn Shah, and Roger Zimmermann. 2020. Keyphrase extraction as sequence labeling using contextualized embeddings. In Advances in Information Retrieval, pages 328–335, Cham. Springer International Publishing. find
  48. Abu Bakr Soliman, Kareem Eissa, and Samhaa R. El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. Arabic Computational Linguistics. find
  49. Dima Suleiman and Arafat Awajan. 2017. Bag-of-concept based keyword extraction from Arabic documents. ICIT 2017 - 8th International Conference on Information Technology, Proceedings, pages 863– 869. find
  50. Dima Suleiman, Arafat A. Awajan, and Wael al Etaiwi. 2019a. Arabic text keywords extraction using word2vec. In 2019 2nd International Conference on new Trends in Computing Sciences (ICTCS), pages 1–7. find
  51. Dima Suleiman, Arafat A. Awajan, and Wael Al Etaiwi. 2019b. Arabic Text Keywords Extraction using Word2vec. 2019 2nd International Conference on New Trends in Computing Sciences, ICTCS 2019 -Proceedings. find
  52. Si Sun, Chenyan Xiong, Zhenghao Liu, Zhiyuan Liu, and Jie Bao. 2020. Joint keyphrase chunking and salience ranking with bert. CoRR 2020, abs/2004.13639. ↗
  53. Yuxiang Zhang, Huan Liu, Suge Wang, W.H. Ip, Fan Wei, and Chunjing Xiao. 2020. Automatic keyphrase extraction using word embeddings. Soft Computing, 24:1–16. ↗
  54. Jing Zhao and Yuxiang Zhang. 2019. Incorporating linguistic constraints into keyphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5224– 5233, Florence, Italy. Association for Computational Linguistics. find
  55. Xian Zu, Fei Xie, and Xiaojian Liu. 2020. Graph-based keyphrase extraction using word and document em beddings. In 2020 IEEE International Conference on Knowledge Graph (ICKG), pages 70–76. find
  56. Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alharbi-2022-arabic-keyphrase-extraction-enhancing-deep/