Elamur Research⌕
☰
home / papers / 2021 / SiPOS: A Benchmark Dataset for Sindhi Part-of-Speech Tagging

SiPOS: A Benchmark Dataset for Sindhi Part-of-Speech Tagging

Summary

Sindhi is a morphologically rich language but has very few resources. Until now it has had no open and sufficiently large annotated corpus for part-of-speech (POS) tagging: earlier datasets are either too small or not publicly available. The authors created SiPOS, a gold-standard dataset built from Sindhi news texts written in the Perso-Arabic script. The texts were first cleaned of noise and tokenized. Then two native speakers with a background in linguistics annotated them manually in Doccano: each token received a language-specific tag and the corresponding universal UPOS category. Inter-annotator agreement was measured with Cohen's Kappa. To validate the resource, the authors built baselines: CRF, and BiLSTM in variants with a CRF decoder and with self-attention. They compared four types of representations: pretrained GloVe and fastText, character-level representations from a BiLSTM encoder, and their combination with word-level ones. The best accuracy, 96.25%, was achieved by the BiLSTM-Attention-CRF model with joint word-character representations. The dataset is open, so it becomes a common point of comparison for future Sindhi taggers.

Key points

  • Size of SiPOS: 293,684 tokens and 6,784 sentences from 1,021 articles of the newspapers Kawish (563 articles) and Awami-Awaz (458). The annotation has 16 categories. The most frequent tag is noun, followed by preposition and verb.
  • Annotation quality: two annotators labeled the data in Doccano; Cohen's Kappa is 0.872. In ambiguous cases they consulted an online Sindhi thesaurus, and the project supervisor monitored adherence to the guidelines.
  • Representations: GloVe and fastText (300 dimensions) were trained on an unannotated Sindhi corpus of over 61M words. Character-level representations are learned for the specific task via a BiLSTM.
  • Experimental observations: the CRF decoder works better than softmax, and self-attention improves accuracy in all configurations. Joint word-character representations are the best, fastText outperforms GloVe, and character-level models come close to fastText.
  • Context: previously, Sindhi had a 33K-token corpus (Mahar and Memon, 2010), a closed 100K-word dataset, a 44K-token Devanagari corpus, and a 100-sentence Latin-script corpus. SiPOS is published on GitHub (AliWazir/SiPOS-Dataset).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cited by

1 paper of our library cites it (of the 140673 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ali2021sipos,
  title = {{SiPOS: A Benchmark Dataset for Sindhi Part-of-Speech Tagging}},
  author = {Ali, Wazir and Xu, Zenglin and Kumar, Jay},
  booktitle = {Proceedings of the Student Research Workshop Associated with RANLP 2021},
  year = {2021},
  doi = {10.26615/issn.2603-2821.2021\_004},
  url = {https://doi.org/10.26615/issn.2603-2821.2021_004}
}

References

36 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wazir Ali Jamro. 2017. Sindhi language processing: A survey. In 2017 International Conference on Innovations in Electrical Engineering and Computational Technologies (ICIEECT), pages 1–8. IEEE. find
  2. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In 12th U SEN IX Symposium on Operating Systems Design and Implementation (OSDI) 16),… find
  3. Yaozong Jia and Xiaopan Ma. 2019. Attention in character-based BiLSTM-CRF for Chinese named entity recognition. In In Proceedings of the 4th International Conference on Mathematics and Artificial Intelligence, pages 1–4. find
  4. Wazir Ali, Jay Kumar, Junyu Lu, and Zenglin Xu. 2019. Word embedding based new corpus for low-resourced language: Sindhi. arXiv preprint arXiv:1911.12579. — [arXiv:1911.12579](https://arxiv.org/abs/1911.12579) find
  5. Katharina Kann, Johannes Bjerva, Isabelle Augenstein, Barbara Plank, and Anders Søgaard. 2018. Character-level supervision for low-resource pos tagging. In Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP, pages 1–11. find
  6. Wazir Ali, Junyu Lu, and Zenglin Xu. 2020. SiNER: A large dataset for Sindhi named entity recognition. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 2946–2954. European Language Resources Association. find
  7. Wahab Khan, Ali Daud, Khairullah Khan, Jamal Abdul Nasir, Mohammed Basheri, Naif Aljohani, and Fahd Saleh Alotaibi. 2019. Part of speech tagging in Urdu: Comparison of machine and deep learning approaches. IEEE Access, 7:38918–38936. find
  8. Monojit Choudhury Tanmoy Bhattacharya Pushpak Bhattacharyya Girish Nath Jha S. Rajendran K. Sara-vanan L. Sobha Baskaran Sankaran, Kalika Bali and K.V. Subbarao. 2008. A common parts-of-speech tagset framework for Indian languages. In Proceedings of the Sixth International Conference on Language Res… find
  9. D Kingma and J Ba. 2015. Adam: A method for stochastic optimization. San Diego. find
  10. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. find
  11. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with Qian Liu, He-Yan Huang, Yang Gao, Xiaochi Wei, Yuxin Tian, and Luyang Liu. 2018. Task-oriented word embedding for text classification. In Proceedings of the 27th International Conference on Computati… find
  12. Association for Computational Linguistics, pages 1756–1765. find
  13. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2089– 2096. find
  14. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064–1074. find our card
  15. Mutee U Rahman. 2010. Towards Sindhi corpus construction. In Proceedings of the Conference on Language and Technology. find
  16. Javed Ahmed Mahar and Ghulam Qadir Memon. 2010. Sindhi part of speech tagging system using wordnet. International Journal of Computer Theory and Engineering, 2(4):538. find
  17. Mutee U Rahman, Tafseer Ahmed, and Muhammad Shaheer Memon. 2020. Development of annotated corpus resources of Sindhi. LANGUAGE & TECHNOLOGY, page 49. find
  18. Mitchell P Marcus and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The penn treebank. Computational Linguistics, 19(2). find our card
  19. Gajanan Rane, Nilesh Joshi, Geetanjali Rane, Hanu-mant Redkar, Malhar Kulkarni, and Pushpak Bhattacharyya. 2020. Part-of-speech annotation challenges in marathi. In Proceedings of the WILDRE5– 5th Workshop on Indian Language Data: Resources and Evaluation, pages 1–6. find
  20. Andrew Matteson, Chanhee Lee, Youngbum Kim, and Heui-Seok Lim. 2018. Rich character-level information for korean morphological analysis and part-of-speech tagging. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2482–2492. find
  21. Nils Reimers and Iryna Gurevych. 2017. Optimal hyperparameters for deep lstm-networks for sequence labeling tasks. arXiv preprint arXiv:1707.06799. — [arXiv:1707.06799](https://arxiv.org/abs/1707.06799) find
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  23. Cicero D Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31st international conference on machine learning (ICML-14), pages 1818–1826. find
  24. Raveesh Motlani, Harsh Lalwani, Manish Shrivastava, and Dipti Misra Sharma. 2015. Developing part-of-speech tagger for a resource poor language: Sindhi. In Proceedings of 7th Conference on Language and Technology, Poznan, Poland. find
  25. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673–2681. find
  26. Yan Shao, Christian Hardmeier, Jörg Tiedemann, and Joakim Nivre. 2017. Character-based joint segmentation and POS tagging for Chinese using bidirectional RNN-CRF. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, pages 173–183. find
  27. Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Ya-sufumi Taniguchi, and Xu Liang. 2018. doccano: Text annotation tool for human. find
  28. Waqar Ali Narejo and Javed Ahmed Mahar. 2016. Morphology: Sindhi morphological analysis for natural language processing applications. In 2016 International Conference on Computing, Electronic and Electrical Engineering (ICE Cube), pages 27–31. IEEE. find
  29. T Shen, T Zhou, G Long, J Jiang, and C Zhang. 2018. Bi-directional block self-attention for fast and memory-efficient sequence modeling. In Proceedings of the International Conference on Representation Learning. find
  30. JD Oad. 2012. Implementing GF resource grammar for Sindhi language. Doctor dissertation. Ph.D. thesis, M. Sc. thesis, Chalmers University of Technology, Gothenburg, Sweden. find
  31. Irum Naz Sodhar, Akhtar Hussain Jalbani, Muhammad Ibrahim Channa, and Dil Nawaz Hakro. 2019. Parts of speech tagging of Romanized Sindhi text by applying rule based model. IJCSNS, 19(11):91. find
  32. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. find
  33. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. find
  34. Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Charles Sutton, Andrew McCallum, et al. 2012. An introduction to conditional random fields. Foundations an… find our card
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  36. Meishan Zhang, Nan Yu, and Guohong Fu. 2018. A simple and effective neural model for joint word segmentation and pos tagging. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(9):1528–1538. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ali-2021-sipos-benchmark-dataset-sindhi-part/