Elamur Research⌕
☰
home / papers / 2022 / Establishing a Baseline for Arabic Patents Classification: A Comparison of Twelve Approaches

Establishing a Baseline for Arabic Patents Classification: A Comparison of Twelve Approaches

Paper · 2022 · Taif Omar Al-Omar, Hend Al-Khalifa and Rawan Al-Matham

Summary

Patent offices still assign classification codes by hand, which is slow. Automatic classification has been studied for English, Korean and Chinese, but Arabic had no public labelled corpus. The authors scraped all granted patents from the Saudi Authority of Intellectual Property (SAIP) website and released them as ArPatent. Each patent is labelled with one of the eight top-level International Patent Classification (IPC) sections. Only the title and abstract are used as input.

They then compared twelve setups:

  • SVM classifiers over TF-IDF, pre-trained AraVec skip-gram vectors, and an in-domain ArPatent-Word2vec model;
  • six fine-tuned Arabic BERT-style models;
  • three weighted max-voting ensembles.

ARBERT was the strongest single system, with a macro-F1 of 66.53%. An ensemble of ARBERT, CAMeL-MSA and QARiB had the best accuracy (70.49%) but a lower macro-F1 (64.52%). The authors trace this to the three models making very similar errors. The very small class D was mostly misclassified by every model. The work gives Arabic patent classification its first open dataset and reference numbers to compare against.

Key points

  • ArPatent dataset: 9,772 patents were scraped with Selenium (about 413 MB, 1.58M words), leaving 9,765 after cleaning. The classes are heavily skewed: section C has 2,783 patents, while section D has only 61.
  • Pre-trained models tested: CAMeL-MSA, CAMeL-MIX, ARBERT, MARBERTv2, AraBERTv2 and QARiB.
  • Fine-tuning setup:
    • learning rate 2e-5 with Adam, dropout 0.1, batch size 32, 3 epochs;
    • inputs truncated at 350 tokens;
    • stratified 80/10/10 split for training, validation and test.
  • Macro-F1 by model:
    • ARBERT 66.53, CAMeL-MSA 59.27, QARiB 57.95, CAMeL-MIX 56.81, SVM-TF-IDF 56.62;
    • MARBERTv2 51.68 and AraBERTv2 51.74, both near the SVMs with embeddings.
  • Ensembles and in-domain vectors:
    • ENSEMBLE-1 used weights 1/0.5/0.5, with ARBERT given priority. It improved accuracy and precision but not F1.
    • The in-domain ArPatent-Word2vec vectors did not beat simpler baselines.

Interesting ideas

  • I1. Before combining models, the authors compare their confusion matrices and find the errors overlap, which explains why voting gave no F1 gain. Checking error diversity first saves wasted ensembling effort.
  • I2. Macro-F1 is the headline metric because the classes are skewed. Accuracy is reported too, which exposes how ENSEMBLE-1 rose in accuracy while F1 fell.
  • I3. Weighted voting with a lead model at weight 1 and two helpers at 0.5 turns the weaker models into tie-breakers or overriders that only win together. It is a cheap way to encode trust in one model without discarding the others.
  • I4. During Arabic normalization, words written in Latin letters were deliberately kept, on the view that they often carry technical terms. Aggressive cleanup can delete the most informative tokens.
  • I5. An embedding trained on in-domain text did not outperform generic resources, which the authors attribute to it being trained only on short titles and abstracts. A domain resource is only worth having if it is fed enough text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Measure how often the witness parsers fail on the same arcs. Agreement among correlated witnesses should earn less silver-data confidence than agreement among independent ones.
  • I2 (learn): Report macro scores over dependency relations or frame types next to overall accuracy, so that regressions on rare labels are not hidden by frequent ones.
  • I3 (learn): When building silver trees, let the strongest parser win unless the other witnesses jointly disagree, and send those cases to the LLM teacher.
  • I4 (tokens): Make sure the tokenizer and tagger keep foreign-script words, formulas and code-like strings as atomic tokens rather than stripping or splitting them.
  • I5 (world): Before compiling domain-specific verb classes or ontology slices from a small corpus, test them against the general WordNet/Wikidata layer as a negative control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBusiness, Management and Accounting · 32 worksManagement of Technology and Innovation · 6 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Taif Omar Al-Omar, Hend Al-Khalifa and Rawan Al-Matham. 2022. Establishing a Baseline for Arabic Patents Classification: A Comparison of Twelve Approaches. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.26

@inproceedings{alomar2022establishing,
  title = {{Establishing a Baseline for Arabic Patents Classification: A Comparison of Twelve Approaches}},
  author = {Taif Omar Al-Omar, Hend Al-Khalifa and Rawan Al-Matham},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.26},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.26}
}

References

19 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelgawad, L., Kluegl, P., Genc, E., Falkner, S., & Hutter, F. (2020). c In U. Brefeld, E. Fromont, A. Hotho, A. Knobbe, M. Maathuis, & C. Robardet (Eds.), Machine Learning and Knowledge Discovery in Databases (pp. 688–703). Springer International Publishing. https://doi.org/10.1007/978-3-030-46133-1_41 ↗
  2. Harris, C. G., Arens, R., & Srinivasan, P. (2010). Comparison of IPC and USPC classification systems in patent prior art searches. Proceedings of the 3rd International Workshop on Patent Information Retrieval, 27–32. https://doi.org/10.1145/1871888.1871894 ↗
  3. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., & Habash, N. (2021). The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models. Proceedings of the Sixth Arabic Natural Language Processing Workshop, 92–104. https://aclanthology.org/2021.wanlp-1.10 ↗
  4. Abdul-Mageed, M., Elmadany, A., & Nagoudi, E. M. B. (2021). ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 7088–7105. https://doi.org/10.18653/v1/2021.acl-long.551 ↗
  5. International Patent Classification (IPC). (n.d.). Retrieved September 9, 2022, from https://www.wipo.int/classifications/ipc/en/index.ht ml Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, & Younes Samih. (2021). Pre-Training BERT on Arabic Tweets: Practical Considerations. https://arxiv.org/abs/2102.10684 ↗
  6. Lee, J.-S., & Hsiang, J. (2020). Patent classification by fine-tuning BERT language model. World Patent Information, 61, 101965. https://doi.org/10.1016/j.wpi.2020.101965 ↗
  7. Antoun, W., Baly, F., & Hajj, H. (2020). AraBERT: Transformer-based Model for Arabic Language Understanding. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, 9–15. https://aclanthology.org/2020.osact-1.2 ↗
  8. Li, S., Hu, J., Cui, Y., & Hu, J. (2018). DeepPatent: Patent classification with convolutional neural networks and word embedding. Scientometrics, 117(2), 721–744. https://doi.org/10.1007/s11192-018-2905-5 ↗
  9. Aristodemou, L., & Tietze, F. (2018). The state-of-the-art on Intellectual Property Analytics (IPA): A literature review on artificial intelligence, machine learning and deep learning methods for analysing intellectual property (IP) data. World Patent Information, 55, 37–51. https://doi.org/10.1016/j.wpi.2018.07.002 ↗
  10. Eleni Kamateri, Vasileios Stamatis, Konstantinos Diamantaras, & Michail Salampasis. (2022). Automated Single-Label Patent Classification using Ensemble Classifiers. https://arxiv.org/abs/2203.03552 ↗
  11. Fall, C. J., Törcsvári, A., Benzineb, K., & Karetka, G. (2003). Automated categorization in the international patent classification. ACM SIGIR Forum, 37(1), 10–25. https://doi.org/10.1145/945546.945547 ↗
  12. Grawe, M. F., Martins, C. A., & Bonfante, A. G. (2017). find
  13. Automated Patent Classification Using Word Embedding. 2017 16th IEEE International Lim, S., & Kwon, Y. (2016). IPC Multi-label Classification Based on the Field Functionality of Patent Documents. In J. Li, X. Li, S. Wang, J. Li, & Q. Z. Sheng (Eds.), Advanced Data Mining and Applications (pp. 677–691). Springer International Publishing. https://doi.org/10.1007/978-3-319-49586-6_48 ↗
  14. Office, E. P. (n.d.). Cooperative Patent Classification (CPC). Retrieved September 9, 2022, from https://www.epo.org/searching-for-patents/helpful-resources/first-time-here/classification/cpc.html#:~:textThe%20Coopera tive%20Patent%20Classification%20(CPC,%2C%2 0groups%20and%20sub%2Dgroups Risch, J., & Krestel, R. (2019). Domain-specific word embeddings for patent classification. Data Technologies and Applications, 53(1), 108–122. https://doi.org/10.1108/DTA-01-2019-0002 ↗
  15. Sofean, M. (2021). Deep learning based pipeline with multichannel inputs for patent classification. World Patent Information, 66, 102060. https://doi.org/10.1016/j.wpi.2021.102060 ↗
  16. Soliman, A. B., Eissa, K., & El-Beltagy, S. R. (2017). AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP. Procedia Computer Science, 117, 256–265. https://doi.org/10.1016/j.procs.2017.10.117 ↗
  17. Tikk, D., Biro, G., & Törcsvári, A. (2008). A Hierarchical Online Classifier for Patent Categorization [Chapter]. Emerging Technologies of Text Mining: Techniques and Applications; IGI Global. https://doi.org/10.4018/978-1-59904-373-9.ch012 ↗
  18. Xiao, L., Wang, G., & Zuo, Y. (2018). Research on Patent Text Classification Based on Word2Vec and LSTM. 2018 11th International Symposium on Computational Intelligence and Design (ISCID), 01, 71–74. https://doi.org/10.1109/ISCID.2018.00023 ↗
  19. Zhu, H., He, C., Fang, Y., Ge, B., Xing, M., & Xiao, W. (2020). Patent Automatic Classification Based on Symmetric Hierarchical Convolution Neural Network. Symmetry, 12(2), 186; https://doi.org/10.3390/sym12020186 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/omar-2022-establishing-baseline-arabic-patents-classification/