Elamur Research⌕
☰
home / papers / 2026 / Survey of Tools for Manual Linguistic Annotation: Supporting Diversity through Interactive Exploration

Survey of Tools for Manual Linguistic Annotation: Supporting Diversity through Interactive Exploration

Summary

Picking a manual annotation tool is difficult because the landscape is fragmented. Earlier reviews mostly list tools with basic facts and rarely detail their practical capabilities. This is a real obstacle for projects on less-resourced languages, unconventional data or structurally complex tasks. The authors surveyed 21 tools between September and December 2024, focusing on morphosyntactic and multi-word expression annotation as defined by Universal Dependencies and PARSEME. Their questionnaire covered:

  • software properties and data formats,
  • language and script support,
  • annotation layers and the user interface,
  • workflow and analysis functions,
  • UD- and PARSEME-specific editing.

Developer teams answered first, and tool users reached through the UniDive COST Action contributed afterwards. The answers were cleaned, merged and released as an open dataset, along with a web catalogue for filtering and comparing tools by feature. The analysis shows a healthy and largely open ecosystem. However, demanding needs, such as complex structures and language-independent annotation, are covered unevenly. The resource serves as a practical, community-extensible guide for annotation projects across diverse languages.

Key points

  • The survey drew 24 responses about 21 tools; three tools were described twice and their entries merged. Tools covered include INCEpTION, brat, ArboratorGrew, UD Annotatrix, TrEd, ELAN, CoNLLU-Editor and Label Studio.
  • The Google Forms questionnaire had 57 questions in eleven thematic sections and documented over 50 features. UD-specific items included editing of lemmas, UPOS, XPOS, FEATS, multiword tokens, basic and enhanced dependencies, empty nodes, metadata, and UD validation.
  • PARSEME-specific items checked support for discontinuous, nested and overlapping multi-word expressions, plus consistency checking.
  • The results are published as a TSV file (tools in rows, features in columns) in the GitHub repository UniDive/corpus-annotation-tools. The repository also holds one YAML file per tool, a JSON summary and the code for the interactive interface.
  • Users can filter tools by combining feature values and view each tool's supported features highlighted. New tools can be added through a template and pull requests, which are reviewed by the maintainers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4u4be6bbtj8e

@inproceedings{pannitto2026survey,
  title = {{Survey of Tools for Manual Linguistic Annotation: Supporting Diversity through Interactive Exploration}},
  author = {Pannitto, Ludovica and Zor, Kaja Dobrovoljc and Guillaume, Bruno},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4u4be6bbtj8e},
  url = {https://doi.org/10.63317/4u4be6bbtj8e}
}

References

39 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammed AbuOdeh, Long Phan, Ahmed Farouk Zakaria Elshabrawy, and Nizar Habash. 2024. Palmyra 3.0: A user-friendly cloud-based platform for morphology and dependency syntax annotation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Eval… find
  2. Giuseppe Attardi. n.d. Dependency grammar annotator. http://medialab.di.unipi.it/Project/QA/Parser/DgAnnotator/. Software tool. No published paper available. ↗
  3. Kristin Berberich and Ingo Kleiber. 2023. Tools for corpus linguistics. https: Accessed: //corpus-analysis.com/. 2025-10-15. find
  4. Ekaterina Borisova, Raia Abu Ahmad, Leyla Garcia-Castro, Ricardo Usbeck, and Georg Rehm. 2024. Surveying the FAIRness of annotation tools: Difficult to find, difficult to reuse. In Proceedings of the 18th Linguistic Annotation Workshop (LAW-XVIII), pages 29–45, St. Julians, Malta. Association for Co… find
  5. Janez Brank. 2023. Q-CAT corpus annotation tool 1.5. Slovenian language resource repository CLARIN.SI. find
  6. Lynnika Butler and Heather Van Volkinburg. 2007. FieldWorks Language Explorer (FLEx). Technology Review, 1(1):1. find
  7. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. Computational Linguistics, 47(2):255–308. find our card
  8. Kaja Dobrovoljc. 2022. Spoken language treebanks in Universal Dependencies: an overview. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1798–1806, Marseille, France. European Language Resources Association. find
  9. Richard Eckart De Castilho, Jan-Christoph Klie, and Iryna Gurevych. 2024. Integrating INCEp-TION into larger annotation processes. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 110–121, Miami, Florida, USA. Association for Com… find
  10. Richard Eckart de Castilho, Éva Mújdricza-Maydt, Seid Muhie Yimam, Silvana Hartmann, Iryna Gurevych, Anette Frank, and Chris Biemann. 2016. A web-based tool for the integrated annotation of semantic and syntactic structures. In Proceedings of the Workshop on Language Technology Resources and Tools f… find
  11. The COLING 2016 Organizing Committee. find
  12. Virginia Felkner, Jennifer Thompson, and Jonathan May. 2024. GPT is not an annotator: The necessity of human annotation in fairness benchmark construction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14104– 14115, Bangkok,… find
  13. Luke Gessler. 2022a. Closing the NLP gap: Documentary linguistics and NLP need a shared software infrastructure. In Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages, pages 119–126, Dublin, Ireland. Association for Computational Linguistics. find
  14. Luke Gessler. 2022b. MALA: The map of applications for linguistic annotation. Software available from https://github.com/lgessler/mala. ↗
  15. Stefan Th. Gries and Andrea L. Berez. 2017. Linguistic annotation in/for corpus linguistics. Handbook of linguistic annotation, pages 379–409. find
  16. Gaël Guibon, Marine Courtin, Kim Gerdes, and Bruno Guillaume. 2020. When collaborative treebank curation meets graph grammars. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5291–5300, Marseille, France. European Language Resources Association. find
  17. Johannes Heinecke. 2019. ConlluEditor: a fully graphical editor for universal dependencies treebank files. In Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019), pages 87–93, Paris, France. Association for Computational Linguistics. find
  18. Nancy Ide and James Pustejovsky. 2017. Handbook of Linguistic Annotation, 1st edition. Springer Publishing Company, Incorporated. find
  19. SIL International. 2019. Field linguist’s toolbox (version 1.6.4). Computer software. find
  20. Parameswari Krishnamurthy, Mitansh Kayathwal, Chirag Jain, and Anushka Agrawal. 2024. UD Annotator. https://plural.iiit.ac.in/conllu-annotator. Developed under the Plural Project at IIIT Hyderabad. Copyright © 2024 Parameswari Krishnamurthy. All rights reserved. ↗
  21. Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proceedings of the 61st Annual Meeting of the… find our card
  22. Stephen Mayhew and Dan Roth. 2018. Talen: Tool for annotation of low-resource entities. In ACL System Demonstrations. find
  23. Mariana Neves and Jurica Ševa. 2021. An extensive review of tools for manual annotation of documents. Brief. Bioinform., 22(1):146–163. find
  24. Petr Pajas and Peter Fabian. 2019. Tree Editor TrEd. https://ufal.mff.cuni.cz/tred/. Software tool. Last released version 2.5236 (Feb 28, 2019). ↗
  25. Agata Savary, Cherifa Ben Khelil, Carlos Ramisch, Voula Giouli, Verginica Barbu Mititelu, Najet Hadj Mohamed, Cvetana Krstev, Chaya Liebeskind, Hongzhi Xu, Sara Stymne, Tunga Güngör, Thomas Pickard, Bruno Guillaume, Eduard Bejček, Archna Bhatia, Marie Candito, Polona Gantar, Uxoa Iñurrieta, Albert G… find
  26. Agata Savary, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova, and Antoine Doucet. 2017. The PARSEME shared task on automatic identification of verbal multiword expressions. In Proceedings of the 13… find our card
  27. Agata Savary, Daniel Zeman, Verginica Barbu Mititelu, Anabela Barreiro, Olesea Caftanatov, Marie-Catherine de Marneffe, Kaja Dobrovoljc, Gülşen Eryiğit, Voula Giouli, Bruno Guillaume, Stella Markantonatou, Nurit Melnik, Joakim Nivre, Atul Kr. Ojha, Carlos Ramisch, Abigail Walsh, Beata Wójtowicz, and… find
  28. Thomas Schmidt. 2012. EXMARaLDA and the FOLK tools — two toolsets for transcribing and annotating spoken language. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 236–240, Istanbul, Turkey. European Language Resources Association (ELRA). find
  29. Sebastian Schuster and Christopher D. Manning. 2016. Enhanced English Universal Dependencies: An improved representation for natural language understanding tasks. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2371–2378, Portorož, Slovenia.… find our card
  30. Eyal Shnarch, Alon Halfon, Ariel Gera, Marina Danilevsky, Yannis Katsis, Leshem Choshen, Martin Santillan Cooper, Dina Epelboim, Zheng Zhang, Dakuo Wang, Lucy Yip, Liat Ein-Dor, Lena Dankin, Ilya Shnayderman, Ranit Aharonov, Yunyao Li, Naftali Liberman, Philip Levin Slesarev, Gwilym Newton, Shila Of… find
  31. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. brat: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstrations Session at EACL 2012, Avignon, France. Association for Computational Linguistics. find our card
  32. Felix Stollenwerk, Joey Öhman, Danila Petrelli, Emma Wallerö, Fredrik Olsson, Camilla Bengtsson, Andreas Horndahl, and Gabriela Zarzar Gandler. 2023. Text annotation handbook: A practical guide for machine learning projects. find
  33. Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2025. Label Studio: Data labeling software. Open source software available from https://github.com/HumanSignal/label-studio. ↗
  34. Utku Türk, Furkan Atmaca, Şaziye Betül Özateş, Abdullatif Köksal, Balkiz Ozturk Basaran, Tunga Gungor, and Arzucan Özgür. 2019. Turkish treebanking: Unifying and constructing efforts. In Proceedings of the 13th Linguistic Annotation Workshop, pages 166–177, Florence, Italy. Association for Computati… find
  35. Francis M. Tyers, Mariya Sheyanova, and Jonathan North Washington. 2017. UD annotatrix: An annotation tool for Universal Dependencies. In Proceedings of the 16th International Workshop on Treebanks and Linguistic Theories, pages 10–17, Prague, Czech Republic. find
  36. Maarten van Gompel and Martin Reynaert. 2013. Folia: A practical xml format for linguistic annotation – a descriptive and comparative study. Computational Linguistics in the Netherlands Journal, 3:63–81. find
  37. Graham Wilcock. 2009. Introduction to linguistic annotation and text analytics. Springer International Publishing, Cham. find
  38. Peter Wittenburg, Hennie Brugman, Albert Russel, Alex Klassmann, and Han Sloetjes. 2006. ELAN: a professional framework for multimodality research. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Associati… find our card
  39. Shuo Zhang and Amir Zeldes. 2017. Gitdox: A linked version controlled online xml editor for manuscript transcription. In FLAIRS, pages 619– 623. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pannitto-2026-survey-tools-manual-linguistic-annotation/