Elamur Research⌕
☰
home / papers / 2016 / Character-Aware Neural Networks for Arabic Named Entity Recognition for Social Media

Character-Aware Neural Networks for Arabic Named Entity Recognition for Social Media

Paper · 2016 · Mourad Gridach

Summary

The paper addresses named entity recognition (Person, Location, Organisation) for Arabic tweets. This is hard for three reasons: the language has rich morphology, it has no capitalisation cues, and social media mixes Dialectal Arabic with MSA. Earlier Arabic NER systems relied on large gazetteers and hand-built features. Gridach replaces these with a learned pipeline. A character-level bidirectional LSTM builds a sub-word vector for each token. This vector is joined with a pretrained word embedding and passed to a word-level bidirectional LSTM. A CRF layer on top picks the best IOB tag sequence as a whole. The system is trained and tested on the public Twitter NER set of Darwish (2013). An ablation adds each component in turn, and every addition helps. The full model reaches 85.71 F1, about 20 points above the best previous system, without gazetteers. The work shows that, for a morphologically rich and noisy domain, learned character and word representations can stand in for costly lexicon engineering.

Key points

  • Data: the Darwish (2013) Arabic Twitter dataset, annotated with ACE guidelines and IOB tags for LOC, PER and ORG. Training and test tweets come from different time periods (May 2012 vs. November 2011).
  • Ablation (F1):
    • plain B-LSTM: 67.33
    • plus character embeddings: 70.80
    • plus pretrained word embeddings: 80.37
    • plus CRF: 85.71 (P 90.57, R 81.36)
  • Comparison with earlier systems on the same set: Darwish and Gao (2014) scored 65.20 F1 using gazetteers and semi-supervision. Zirikly and Diab (2015) scored 59.59 F1 using many hand-made features.
  • Embeddings: the pretrained Arabic vectors of Zahran et al. (2015) were used. word2vec beat GloVe by 2.46 F1 and beat random initialisation by 10.49 F1.
  • Setup:
    • word embeddings of dimension 100
    • a character BiLSTM with 25 units per direction, giving a 50-dimensional character vector
    • plain mini-batch SGD, which gave better final scores than Adam, RMSProp or Adadelta, even though those converged faster
    • preprocessing that maps all digits to zero and splits sentences at "." and ","

Interesting ideas

  • I1. A small character-level BiLSTM builds a vector for each word from its spelling, which captures affixes and orthographic cues without hand-made rules and helps with unseen or noisy forms.
  • I2. A CRF on top of a neural tagger scores whole tag sequences through a learned transition matrix, so impossible label chains (such as an I-tag right after O) are penalised. Here this alone added over 5 F1.
  • I3. The evaluation adds one component at a time and reports the F1 gain of each, which makes it clear where the improvement really comes from.
  • I4. Comparing pretrained embeddings against random initialisation separates the value of unlabeled-corpus knowledge from the value of the architecture itself.
  • I5. The optimiser that converges fastest is not always the one that generalises best; plain SGD won on final scores here.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): such a character encoder could serve as a fallback for out-of-vocabulary words, typos and social-media spellings in the en tagger and lemmatiser. Its suggestions should be logged as explicit guesses, keeping the step explainable.
  • I2 (tokens): a transition layer like this could enforce consistent multi-word name spans before parsing, and the transition table is directly inspectable.
  • I3 (learn): every new feature in our parsers or taggers could be reported as such a cumulative ladder, with a random-initialisation baseline acting as a negative control.
  • I4 (learn): when testing silver data or teacher tiers, the same "with vs. random/without" contrast would show how much each knowledge source actually contributes.
  • I5 (learn): in perceptron and gparse training runs we should judge optimisers by held-out scores, not by convergence speed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Colorado BoulderBoulder · United States69 works · 51,069 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Mourad Gridach. 2016. Character-Aware Neural Networks for Arabic Named Entity Recognition for Social Media. Proceedings of the 6th Workshop on South and Southeast Asian Natural Language Processing (WSSANLP2016).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gridach2016character,
  title = {{Character-Aware Neural Networks for Arabic Named Entity Recognition for Social Media}},
  author = {Mourad Gridach},
  booktitle = {Proceedings of the 6th Workshop on South and Southeast Asian Natural Language Processing (WSSANLP2016)},
  year = {2016},
  url = {https://aclanthology.org/W16-3703/}
}

References

34 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bogdan Babych and Anthony Hartley. 2003. Improving Machine Translation Quality with Automatic Named Entity Recognition, Proceedings of EACL, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based dependency parsing by modeling characters instead of words with LSTMs, Proceedings of EMNLP, Lisbon. Association for Computational Linguistics. find
  2. Mohit Bansal Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing, Proceedings of ACL, Baltimore. Association for Computational Linguistics.
  3. Mikael Boden. 2002. A guide to recurrent neural networks and backpropagation, Technical report. find
  4. Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns, arXiv preprint arXiv:1511.08308 ↗
  5. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation, Proceedings of EMNLP, Doha. Association for Computational Linguistics. ↗
  6. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A Character-level Decoder without Explicit Segmentation for Neural Machine Translation, Proceedings of ACL, Berlin. Association for Computational Linguistics Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koran Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (Almost) from Scratch, Journal of Machine Learning Research,12(1):2493- 2537.
  7. Kareem Darwish. 2013. Named entity recognition using cross-lingual resources: Arabic as an example, Proceedings of ACL, Sofia, Bulgaria. Association for Computational Linguistics.
  8. Kareem Darwish and Wei Gao. 2014. Simple effective microblog named entity recognition: Arabic as an example, Proceedings of the Ninth International Conference on Language Resources and Evaluation, Reykjavik, Iceland.
  9. Cìcero dos Santos and Maìra Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts, Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics. Technical Papers, Dublin, Ireland Association for Computational Linguistics.
  10. Sergio Ferrndez, Antonio Toral, scar Ferrndez, Antonio Ferrndez, and Rafael Muoz. 2007. Applying wikipedias multilingual knowledge to crosslingual question answering, Zoubida Kedad, Nadira Lammari, Elisabeth Mtais, Farid Meziane, and Yacine Rezgui, editors, Reykjavik, Iceland. Springer. find
  11. Christoh Goller and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure, Proceedings of the International Conference on Neural Networks, Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM networks, Proceedings of IJCNN. find
  12. Alex Graves. 2013. Generating sequences with recurrent neural networks, arXiv preprint Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. 2012. Coursera, Lecture 6e: rmsprop: divide the gradient by a running average of its recent magnitude, Neural Networks for Machine Learning, Sepp Hochreiter and Jürgen Schmidhuber. 9(8):1735–1780. 1997. Long short-term memory, Neural Computation, Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition, arXiv preprint Fei Huang, Arun Ahuja, Doug Downey, Yi Yang, Yuhong Guo, and Alexander Yates. 2014. Learning representations for weakly supervised natural language processing tasks, Computational Linguistics, 40(1).
  13. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging, arXiv preprint arXiv:1508.01991 ↗
  14. Jun’ichi Kazama and Kentaro Torisawa. 2007. Exploiting Wikipedia as external knowledge for named entity recognition, Proceedings of EMNLP - CoNLL, Prague, Czech Republic. Association for Computational Linguistics.
  15. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2015. models, CoRR, abs/1508.06615 Character-aware neural language Diederik Kingma and Jimmy Ba. arXiv:1412.6980. 2014. Adam: A method for stochastic optimization, arXiv preprint John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data, Proceedings of ICML. ↗
  16. Wang Ling, Tiago Luıs, Luıs Marujo, Ramon F. Astudillo, Silvio Amir, Chris Dyer, Alan W. Black, and Isabel Trancoso. 2015. Finding function in form: Compositional character models for open vocabulary word representation, Proceedings of EMNLP - CoNLL, Lisbon, Portugal. Association for Computational Linguistics. ↗
  17. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better Word Representations with Recursive Neural Networks for Morphology, Proceedings of CoNLL, Sofia, Bulgaria. Association for Computational Linguistics.
  18. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models, Proceedings of ACL, Berlin, Germany. Association for Computational Linguistics.
  19. Tomàš Mikolov, Stefan Kombrink, Lucàš Burget, Jan Cernocký, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model, Proceedings of ICASSP. find
  20. Tomàš Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality, Proceedings of NIPS. ↗
  21. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification, Journal of Linguisticae Investigationes, 30(1). find
  22. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams 1986. Learning Internal Representations by Error Propagation, Symposium on Parallel and Distributed Processing. find
  23. Semiocast. 2012. Geolocation analysis of Twitter accounts and tweets by Semiocast. find
  24. Khaled Shaalan and Mai Oudah. 2014. Information Science, 40(1):67–87. A hybrid approach to Arabic named entity recognition, Journal of Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank, Proceedings of EMNLP, Seattle, USA. Association for Computational Linguistics.
  25. Radu Soricut and Franz Och. 2015. Unsupervised Morphology Induction Using Word Embeddings, Proceedings of the NAACL-HLT, Denver, Colorado. Association for Computational Linguistics. find
  26. Ilya Sutskever, Orion Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks, Proceedings of NIPS, Montreal, Canada.
  27. Paul Thompson and Christopher C. Dozier. 1997. Name searching and information retrieval, Proceedings of EMNLP, Association for Computational Linguistics. find
  28. Hiroyuki Toda and Ryoji Kataoka. 2005. A Search Result Clustering Method using Informatively Named Entities, Proceedings of the 7th annual ACM international workshop on Web information and data management (WIDM), ACM Press. find
  29. Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio 2010. Word representations: A simple and general method for semi-supervised learning, Proceedings of ACL, Uppsala, Sweden. Association for Computational Linguistics.
  30. Yijun Xiao and Kyunghyun Cho 2016. Efficient character-level document classification by combining convolution and recurrent layers, arXiv preprint arXiv:1602.00367 ↗
  31. Mohamed A. Zahran, Ahmed Magooda, Ashraf Y. Mahgoub, Hazem Raafat, Mohsen Rashwan, and Amir Atyia. 2015. Word Representations in Vector Space and their Applications for Arabic, the Proceedings of CICLing, Cairo, Egypt. Springer. find
  32. Matthew D. Zeiler 2012. ADADELTA: an adaptive learning rate method, CoRR, abs/1212.5701 ↗
  33. Xiang Zhang, Junbo Zhao, and Yann LeCun 2015. Character-level convolutional networks for text classification, the Proceedings of NIPS, Montreal, Canada.
  34. Ayah Zirikly and Mona Diab. 2015. Named Entity Recognition for Arabic Social Media, NAACL-HLT, Denver, Colorado. Association for Computational Linguistics. Proceedings of the

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gridach-2016-character-aware-neural-networks-arabic/