Elamur Research⌕
☰
home / papers / 2013 / WebAnno: A Flexible, Web-based and Visually Supported System for Distributed Annotations

WebAnno: A Flexible, Web-based and Visually Supported System for Distributed Annotations

Summary

Statistical NLP depends on annotated training data, yet annotation tools have often been built for a single task, tied to proprietary formats, or required local installation. The authors present WebAnno, an open-source (Apache 2.0) annotation platform that runs entirely in the browser. It reuses the visualization front end of the brat tool but replaces brat's server side with a Java back end built on Apache Wicket. The new back end adds:

  • project and user management, with tagsets configured in the browser;
  • a separate copy of each document per annotator;
  • a curation view for resolving disagreements;
  • progress monitoring with inter-annotator agreement;
  • a bridge to the CrowdFlower crowdsourcing service.

Internally, annotations follow the DKPro Core UIMA type system, and pluggable DKPro Core readers and writers handle import and export. The described version supports part-of-speech, named entity, dependency and co-reference annotation, and a paging mechanism keeps the editor responsive on long documents. The contribution is a system rather than an experimental result. It lowers the setup cost of a quality-controlled, multi-annotator project and makes annotation accessible to both trained and untrained workers.

Key points

  • The paper describes WebAnno v0.3.0. It builds on brat's visualization but reports that brat itself becomes slow beyond about 100 sentences, has limited file formats, and cannot configure tagsets or layer display through the web.
  • Supported input formats are plain text, CoNLL, TCF and UIMA XMI. Annotations are stored as binary-serialized UIMA CAS objects in the file system, while projects, tagsets and users are kept in an SQL database.
  • There are three main roles: project manager/administrator, annotator and curator. Documents move through the states in progress, complete (locked for the annotator) and adjudicated.
  • The monitoring component computes agreement with Kappa and Tau. It also shows a matrix of annotators versus documents marking completed and adjudicated work.
  • Crowdsourcing is handled by a separate module using CrowdFlower's API. It splits documents into small pieces, defines test items and aggregates the answers, which then appear as a "virtual annotator." At this stage, only named entity templates are available.

Interesting ideas

  • I1. Crowd results as a virtual annotator. Aggregated crowd output is treated as one more annotator. This lets the same agreement and curation machinery handle crowd and expert labels without special code, which suits any pipeline that mixes label sources.
  • I2. One copy per annotator, merged later. Each annotator works on a private copy and a curator merges the results afterwards. This avoids concurrent-editing conflicts and keeps every judgment available for measuring agreement.
  • I3. Paging by sentences. Showing only a limited number of sentences at a time makes interface speed independent of document length. This is a simple fix for any browser-based viewer of long texts.
  • I4. Showing only the relevant layers. Annotators see and edit only the layers they are working on. This reduces both visual clutter and rendering cost, and is a useful design choice for multi-layer corpora such as treebanks with NER.
  • I5. Disagreement-driven navigation in curation. The curation view lists only the sentences where annotators differ, so curators can jump straight to conflicts. This makes adjudicating large corpora much cheaper.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition · Part-of-speech tagging

Cited by

115 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 109 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yimam2013webanno,
  title = {{WebAnno: A Flexible, Web-based and Visually Supported System for Distributed Annotations}},
  author = {Yimam, Seid Muhie and Gurevych, Iryna and Castilho, Richard Eckart de and Biemann, Chris},
  booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations},
  year = {2013},
  url = {https://aclanthology.org/P13-4001/}
}

References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Biemann. 2013. Creating a system for lexical substitutions from scratch using crowdsourcing. Lang. Resour. Eval., 47(1):97–122, March. find
  2. Kalina Bontcheva, Hamish Cunningham, Ian Roberts, and Valentin Tablan. 2010. Web-based collaborative corpus annotation: Requirements and a framework implementation. In New Challenges for NLP Frameworks workshop at LREC-2010, Malta. find
  3. Jean Carletta. 1996. Assessing agreement on classification tasks: the kappa statistic. In Computational Linguistics, Volume 22 Issue 2, pages 249–254. find our card
  4. Richard Eckart de Castilho and Iryna Gurevych. 2009. DKPro-UGD: A Flexible Data-Cleansing Approach to Processing User-Generated Discourse. In Online-proceedings of the First French-speaking meeting around the framework Apache UIMA, LINA CNRS UMR 6241 - University of Nantes, France. find
  5. David Ferrucci and Adam Lally. 2004. UIMA: An Architectural Approach to Unstructured Information Processing in the Corporate Research Environment. In Journal of Natural Language Engineering 2004, pages 327–348. find
  6. Ulrich Heid, Helmut Schmid, Kerstin Eckart, and Erhard Hinrichs. 2010. A Corpus Representation Format for Linguistic Web Services: the D-SPIN Text Corpus Format and its Relationship with ISO Standards. In Proceedings of LREC 2010, Malta. find
  7. Boci Lin, Yan Chen, Xu Chen, and Yingying Yu. 2012. Comparison between JSON and XML in Applications Based on AJAX. In Computer Science & Service System (CSSS), 2012, Nanjing, China. find
  8. Thomas Morton and Jeremy LaCivita. 2003. WordFreak: an open tool for linguistic annotation. In Proceedings of NAACL-2003, NAACL-Demonstrations ’03, pages 17–18, Edmonton, Canada. find
  9. Christoph Müller and Michael Strube. 2006. Multi-level annotation of linguistic data with MMAX2. In S. Braun, K. Kohn, and J. Mukherjee, editors, Corpus Technology and Language Pedagogy: New Resources, New Tools, New Methods, pages 197–214. Peter Lang, Frankfurt a.M., Germany. find
  10. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan McDonald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. find
  11. The CoNLL 2007 Shared Task on Dependency Parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932, Prague, Czech Republic. find our card
  12. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. brat: a Web-based Tool for NLP-Assisted Text Annotation. In Proceedings of the Demonstrations at EACL-2012, Avignon, France. find our card
  13. Qingling Wang, Qin Liu, Na Li, and Yan Liu. 2008. An Automatic Approach to Reengineering Common Website with AJAX. In 4th International Conference on Next Generation Web Services Practices, pages 185–190, Seoul, South Korea. find
  14. Aobo Wang, Cong Duy Vu Hoang, and Min-Yen Kan. 2013. Perspectives on Crowdsourcing Annotations for Natural Language Processing. In Language Resources And Evaluation, pages 9–31. Springer Netherlands. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yimam-2013-webanno-flexible-web-based-visually/