Elamur Research⌕
☰
home / papers / 2016 / Arabic Named Entity Recognition using Word Representations

Arabic Named Entity Recognition using Word Representations

Paper · 2016 · Ismail El Bazi and Nabil Laachfoubi

Summary

Supervised named entity recognition for Arabic is held back by the small amount of hand-annotated data. The authors ask whether word representations learned from large unlabeled corpora, which had already improved English NER, help Arabic in the same way. They keep one CRF tagger (CRFsuite) fixed and only add features derived from representations. They compare three embedding learners (SKIP-gram, CBOW, GloVe) and six ways of turning representations into features:

  • Brown clusters
  • raw dense vectors
  • binarized vectors, using three thresholding schemes
  • sparse overcomplete vectors (SE and NNSE)
  • k-means clusters of embeddings
  • per-class distributional prototypes

Training and testing use the AQMAR corpus. Brown clustering is the best single feature family. Among embedding-based features, clustered embeddings work best and prototypes come second. Combining Brown clusters with embedding features improves F1 by nearly 10% over the baseline. The study shows that cheap, discretized unsupervised features are a practical fix for data sparsity in NER for a morphologically rich, low-resource language.

Key points

  • Model: a linear-chain CRF implemented with CRFsuite. The representation features are added on top of a baseline feature set, and the model itself does not change.
  • Embeddings were trained with SKIP-gram, CBOW and GloVe. The Brown clusters (examples in Table I) were induced from Arabic Wikipedia dumps.
  • Binarization comes in three variants:
    • Faruqui et al.'s method (A);
    • Guo et al.'s mean-based method (B);
    • the authors' own median-based variant (C), which maps each dimension to positive, negative or zero.
  • Prototypes are the top-m words per BIO entity class in the AQMAR training set, ranked by NPMI. A word receives a prototype as a feature when its cosine similarity to that prototype exceeds a threshold of about 0.5.
  • Ranking: Brown clustering is best overall. Among embedding features, clustering embeddings rank first and prototypes second. Brown clusters plus embedding features give a gain of nearly 10% in F1 over the baseline.

Interesting ideas

  • I1. Dense vectors work poorly in a linear model, so cluster them with k-means at several values of k and use the cluster IDs as discrete features, giving coarse and fine granularity together.
  • I2. Distributional prototypes: pick the words most associated with each label in labeled data (by NPMI), then tag any word whose vector is close enough to a prototype with that prototype as a feature.
  • I3. Brown clustering gives every word a unique path in a binary tree, which supplies a hierarchical class signal learned without supervision. In this study it beat every embedding-based feature family.
  • I4. Three-valued binarization keeps only the strongly positive or strongly negative dimensions, cutting at the mean or median of each sign, and zeros out the rest.
  • I5. The experiment holds the learner fixed and varies only the feature-construction method, so the comparison isolates the value of each representation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the tagger could take multi-granularity cluster IDs as sparse features for rare or unknown words, and each decision stays explainable as "this word falls in cluster 37 at k=500".
  • I2 (world): unseen names could get a provisional entity type with a readable justification ("similar to Cairo, a known LOC") before the knowledge graph confirms or rejects it.
  • I3 (tree): the perceptron parser could add Brown-path features to back off on sparse lexical features, which is a cheap way to make it more robust on out-of-domain words.
  • I4 (learn): this is a simple, testable way to turn transformer or embedding outputs from gparse into symbolic features for linear components, and the mean and median cuts can be compared against each other.
  • I5 (learn): ablations of new feature sources could follow the same fixed-model protocol, run alongside the negative controls, so that gains are attributed to the features and not to model changes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Université Hassan 1er.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Ismail El Bazi and Nabil Laachfoubi. 2016. Arabic Named Entity Recognition using Word Representations. arXiv:1804.05630 · International Journal of Computer Science and Information Security (IJCSIS), Vol. 14, No. 8, August 2016.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1804.05630 · arXiv 1804.05630

@misc{bazi2016arabic,
  title = {{Arabic Named Entity Recognition using Word Representations}},
  author = {Ismail {El Bazi} and Nabil Laachfoubi},
  year = {2016},
  doi = {10.48550/arxiv.1804.05630},
  eprint = {1804.05630},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1804.05630}
}

References

36 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Liang, “Semi-supervised learning for natural language,” Massachusetts Institute of Technology, 2005. find
  2. J. Turian, L. Ratinov, and Y. Bengio, “Word representations: a simple and general method for semi-supervised learning,” in Proceedings of the 48th annual meeting of the association for computational linguistics, 2010, pp. 384–394.
  3. J. Turian, L. Ratinov, Y. Bengio, and D. Roth, “A preliminary evaluation of word representations for named-entity recognition,” in NIPS Workshop on Grammar Induction, Representation of Language and Language Learning, 2009, pp. 1–8. find
  4. A. Passos, V. Kumar, and A. McCallum, “Lexicon Infused Phrase Embeddings for Named Entity Resolution,” in Proceedings of the Eighteenth Conference on Computational Natural Language Learning, CoNLL 2014, Baltimore, Maryland, USA, June 26-27, 2014, 2014, pp. 78–86.
  5. J. Guo, W. Che, H. Wang, and T. Liu, “Revisiting Embedding Features for Simple Semi-supervised Learning.,” in EMNLP, 2014, pp. 110–120. find
  6. A. Abdul-Hamid and K. Darwish, “Simplified Feature Set for Arabic Named Entity Recognition,” in Proceedings of the 2010 Named Entities Workshop, 2010, pp. 110–115.
  7. I. El Bazi and N. Laachfoubi, “A comparative study of Named Entity Recognition for Arabic using ensemble learning approaches,” in 2015 IEEE/ACS 12th International Conference of Computer Systems and Applications (AICCSA), 2015, pp. 1–6. find
  8. K. Shaalan and M. Oudah, “A hybrid approach to Arabic named entity recognition,” Journal of Information Science, vol. 40, no. 1, pp. 67–87, 2014. ↗
  9. Y. Benajiba, P. Rosso, and J. M. Bened𝚤ruiz, “Anersys: An arabic named entity recognition system based on maximum entropy,” in International Conference on Intelligent Text Processing and Computational Linguistics, 2007, pp. 143–153. CBOW GloVe F1 F1 55,45 55,45 56,19 56,41 55,36 55,51 55,58 55,79 56,87 55,26 56,55 56,3 55,54 56,48 60,28 60,98 58.9 56.86 61,05 60,2 62,45 62,45 63,55 66,57 64,39 66,23 65,36 66,57 https://sites.google.com/site/ijcsis/ISSN 1947-5500 International Journal of Computer Science and Information Security (IJCSIS), Vol. 14, No. 8, August 2016 ↗
  10. N. Laachfoubi and others, “RENA: A Named Entity Recognition System for Arabic,” in International Conference on Text, Speech, and Dialogue, 2015, pp. 396–404. find
  11. Y. Benajiba, M. Diab, and P. Rosso, “Arabic named entity recognition using optimized feature sets,” in In Proc. of EMNLP’08, 2008, pp. 284–293.
  12. A. Zirikly and M. Diab, “Named entity recognition for dialectal arabic,” ANLP 2014, p. 78, 2014. find
  13. A. Zirikly and M. Diab, “Named entity recognition for arabic social media,” in Proceedings of naacl-hlt, 2015, pp. 176–185.
  14. O. Täckström, R. McDonald, and J. Uszkoreit, “Cross-lingual word clusters for direct transfer of linguistic structure,” in Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies, 2012, pp. 477–487.
  15. F. Huang and A. Yates, “Distributional representations for handling sparsity in supervised sequence-labeling,” in Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, 2009, pp. 495–503. find
  16. T. Koo, X. Carreras Pérez, and M. Collins, “Simple semi-supervised dependency parsing,” in 46th Annual Meeting of the Association for Computational Linguistics, 2008, pp. 595–603.
  17. P. F. Brown, P. V. Desouza, R. L. Mercer, V. J. D. Pietra, and J. C. Lai, “Class-based n-gram models of natural language,” Computational linguistics, vol. 18, no. 4, pp. 467–479, 1992. ↗
  18. D. Lin and X. Wu, “Phrase clustering for discriminative learning,” in Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, 2009, pp. 1030–1038.
  19. A. Clark, “Combining distributional and morphological information for part of speech induction,” Proceedings of 10th EACL, pp. 59–66, 2003.
  20. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “A neural probabilistic language model,” journal of machine learning research, vol. 3, no. Feb, pp. 1137–1155, 2003.
  21. S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman, “Indexing by latent semantic analysis,” Journal of the American society for information science, vol. 41, no. 6, p. 391, 1990. find
  22. K. Lund and C. Burgess, “Producing high-dimensional semantic spaces from lexical co-occurrence,” Behavior Research Methods, Instruments, & Computers, vol. 28, no. 2, pp. 203–208, 1996. ↗
  23. R. Lebret and R. Collobert, “Word Embeddings through Hellinger PCA,” in Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, 2014, pp. 482–490. find
  24. J. Pennington, R. Socher, and C. D. Manning, “Glove: Global Vectors for Word Representation.,” in EMNLP, 2014, vol. 14, pp. 1532–43.
  25. R. Collobert and J. Weston, “A unified architecture for natural language processing: Deep neural networks with multitask learning,” in Proceedings of the 25th international conference on Machine learning, 2008, pp. 160–167. find
  26. A. Mnih and G. E. Hinton, “A scalable hierarchical distributed language model,” in Advances in neural information processing systems, 2009, pp. 1081–1088. find
  27. T. Mikolov and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in neural information processing systems, 2013. ↗
  28. R. Al-Rfou, B. Perozzi, and S. Skiena, “Polyglot: Distributed word representations for multilingual nlp,” arXiv preprint arXiv:1307.1662, 2013. ↗
  29. M. Yu, T. Zhao, D. Dong, H. Tian, and D. Yu, “Compound Embedding Features for Semi-supervised Learning.,” in HLT-NAACL, 2013, pp. 563–568. find
  30. M. Faruqui, Y. Tsvetkov, D. Yogatama, C. Dyer, and N. Smith, “Sparse overcomplete word vector representations,” arXiv preprint arXiv:1506.02004, 2015. ↗
  31. B. Murphy, P. P. Talukdar, and T. M. Mitchell, “Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding.,” in COLING, 2012, pp. 1933–1950. find
  32. A. Fyshe, P. P. Talukdar, B. Murphy, and T. M. Mitchell, “Interpretable semantic vectors from a joint model of brain-and text-based meaning,” in Proceedings of the conference. Association for Computational Linguistics. Meeting, 2014, vol. 2014, p. 489. find
  33. H. Lee, A. Battle, R. Raina, and A. Y. Ng, “Efficient sparse coding algorithms,” in Advances in neural information processing systems, 2006, pp. 801–808. ↗
  34. G. Bouma, “Normalized (pointwise) mutual information in collocation extraction,” Proceedings of GSCL, pp. 31–40, 2009. find
  35. B. Mohit, N. Schneider, R. Bhowmick, K. Oflazer, and N. A. Smith, “Recall-oriented Learning of Named Entities in Arabic Wikipedia,” in Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, 2012, pp. 162–173.
  36. J. Lafferty, A. McCallum, and F. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” in Proceedings of the eighteenth international conference on machine learning, ICML, 2001, vol. 1, pp. 282– 289. https://sites.google.com/site/ijcsis/ISSN 1947-5500 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bazi-2018-arabic-named-entity-recognition-using/