Elamur Research⌕
☰
home / papers / 2012 / brat: a Web-based Tool for NLP-Assisted Text Annotation

brat: a Web-based Tool for NLP-Assisted Text Annotation

Summary

Building gold-standard corpora is slow and expensive. Most annotation tools of the time were aimed at technical users and offered little beyond the basics. The authors present BRAT, a browser-based annotation environment built on their earlier STAV visualiser. It lets annotators mark text spans, binary relations, n-ary event structures, attributes and free-text notes, all through familiar mouse gestures. It also checks configurable constraints on the annotation schema, offers corpus search and imports output from automatic tools. Its NLP assistance is designed to support human decisions rather than replace them. A machine-learning semantic class disambiguation component narrows the list of candidate types the annotator must choose from. In a controlled experiment, one expert annotated 80 biomedical abstracts with 54 mention types. Assisted ("rapid") mode cut total annotation time by about 15%, almost entirely because choosing a type became faster. The tool is open source under the MIT license, and during development it had already been used to create more than 50,000 annotations. It therefore shows that a usable interface combined with light ML assistance can lower annotation costs.

Key points

  • Rapid mode reduced total annotation time by 15.4% (45:28 → 39:24) and type-selection time by 30.7% (13:49 → 09:35). The two conditions produced nearly identical annotation counts (1,546 normal vs 1,541 rapid).
  • The disambiguation component shrank the choice set from 54 types (19 physical entity, 35 event/process) to an average of 2.88. In earlier work across six corpora, it cut ambiguity by over 75% while keeping the correct class about 99% of the time (Stenetorp et al., 2011a).
  • In 2011, the tool was used to annotate six corpora by four research groups, including the EPI and ID corpora of the BioNLP Shared Task 2011. Other projects covered cancer-biology events, Japanese verb frames and gene-mutation-phenotype relations.
  • Technical design:
    • The client uses XHTML, SVG and jQuery and talks to the server over JSON/AJAX.
    • The server is a stateless Python RESTful back-end supporting CGI or FastCGI, with FastCGI responses under 100 ms.
    • Annotations are stored in a file-based stand-off format.
    • Text handling is fully Unicode.
  • BRAT ships with example data from over 20 corpora in seven languages, including CoNLL NER and multilingual dependency parsing data. It is available at brat.nlplab.org.

Interesting ideas

  • I1. Narrow choices instead of auto-labelling. Rather than pre-filling labels that annotators might accept uncritically, the system only shortens the menu of plausible types. The human still decides every item. This lowers bias from automatic pre-annotation while saving time, which suits any large tagset (e.g., fine-grained NER or UD relations).
  • I2. Validate the schema live. Constraints such as required argument roles and allowed argument types are checked as the annotator works, and violations are highlighted visually. Errors are caught at creation time, not in a later consistency pass, which matters for complex event or relation schemes.
  • I3. Let the server own the data. Every edit is sent to the server immediately, so there is no save button and no merging of divergent copies. This also allows several annotators to work on one document at the same time, which helps small distributed teams.
  • I4. Give every annotation a URI. Each annotation can be linked directly from e-mail, documents or web pages. This makes guideline discussions and error reports concrete, a cheap trick for any collaborative curation workflow.
  • I5. Control for confounds in productivity tests. The experiment used a single experienced annotator and stratified-sampled pairs of 10-document sets with matched statistics. Assisted and unassisted sets alternated so that learning effects would not favour either mode. This design is a reusable template for measuring whether a tool change really speeds up annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

463 papers of our library cite it (of the 140884 whose bibliographies we read); 25 of them have a page here, 438 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{stenetorp2012brat,
  title = {{brat: a Web-based Tool for NLP-Assisted Text Annotation}},
  author = {Stenetorp, Pontus and Pyysalo, Sampo and Topić, Goran and Ohta, Tomoko and Ananiadou, Sophia and Tsujii, Jun'ichi},
  booktitle = {Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2012},
  url = {https://aclanthology.org/E12-2021/}
}

References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Abney. 1991. Parsing by chunks. Principle-based parsing, 44:257–278. find
  2. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X), pages 149–164. find our card
  3. Stuart K. Card, Thomas P. Moran, and Allen Newell. 1983. The psychology of human-computer interaction. Lawrence Erlbaum Associates, Hillsdale, New Jersey. find
  4. Xavier Carreras and Lluı́s Màrquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic Role Labeling. In Proceedings of the 9th Conference on Natural Language Learning, pages 152–164. Association for Computational Linguistics. find our card
  5. Hamish Cunningham, Diana Maynard, Kalina Bontcheva, Valentin Tablan, Niraj Aswani, Ian Roberts, Genevieve Gorrell, Adam Funk, Angus Roberts, Danica Damljanovic, Thomas Heitz, Mark A. Greenwood, Horacio Saggion, Johann Petrak, Yaoyong Li, and Wim Peters. 2011. Text Processing with GATE (Version 6). find
  6. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) program: Tasks, data, and evaluation. In Proceedings of the 4th International Conference on Language Resources and Evaluation, pages 837– 840. find our card
  7. Roy Fielding. 2000. REpresentational State Transfer (REST). Architectural Styles and the Design of Network-based Software Architectures. University of California, Irvine, page 120. find
  8. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. find our card
  9. Jin-Dong Kim, Tomoko Ohta, and Jun’ichi Tsujii. 2008. Corpus annotation for mining biomedical events from literature. BMC Bioinformatics, 9(1):10. find
  10. Jin-Dong Kim, Sampo Pyysalo, Tomoko Ohta, Robert Bossy, Ngan Nguyen, and Jun’ichi Tsujii. 2011. Overview of BioNLP Shared Task 2011. In Proceedings of BioNLP Shared Task 2011 Workshop, pages 1–6, Portland, Oregon, USA, June. Association for Computational Linguistics. find
  11. LDC. 2005. ACE (Automatic Content Extraction) English Annotation Guidelines for Events. Technical report, Linguistic Data Consortium. find
  12. Pascual Martı́nez-Gómez, Germán Sanchis-Trilles, and Francisco Casacuberta. 2011. Online learning via dynamic reranking for computer assisted translation. In Alexander Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, volume 6609 of Lecture Notes in Computer Science, page… find
  13. Joakim Nivre. 2003. An Efficient Algorithm for Projective Dependency Parsing. In Proceedings of the 8th International Workshop on Parsing Technologies, pages 149–160. find our card
  14. Philip V. Ogren. 2006. Knowtator: A protégé plug-in for annotated corpus construction. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Companion Volume: Demonstrations, pages 273–275, New York City, USA,… find
  15. Sampo Pyysalo, Tomoko Ohta, Rafal Rak, Dan Sullivan, Chunhong Mao, Chunxia Wang, Bruno Sobral, Junichi Tsujii, and Sophia Ananiadou. 2012. Overview of the ID, EPI and REL tasks of BioNLP Shared Task 2011. BMC Bioinformatics, 13(suppl. 8):S2. find
  16. David Salgado, Martin Krallinger, Marc Depaule, Elodie Drula, and Ashish V Tendulkar. 2010. Myminer system description. In Proceedings of the Third BioCreative Challenge Evaluation Workshop 2010, pages 157–158. find
  17. Pontus Stenetorp, Sampo Pyysalo, Sophia Ananiadou, and Jun’ichi Tsujii. 2011a. Almost total recall: Semantic category disambiguation using large lexical resources and approximate string matching. In Proceedings of the Fourth International Symposium on Languages in Biology and Medicine. find
  18. Pontus Stenetorp, Goran Topić, Sampo Pyysalo, Tomoko Ohta, Jin-Dong Kim, and Jun’ichi Tsujii. 2011b. find
  19. BioNLP Shared Task 2011: Supporting Resources. In Proceedings of BioNLP Shared Task 2011 Workshop, pages 112–120, Portland, Oregon, USA, June. Association for Computational Linguistics. find
  20. Beth M. Sundheim. 1996. Overview of results of the MUC-6 evaluation. In Proceedings of the Sixth Message Understanding Conference, pages 423– 442. Association for Computational Linguistics. find
  21. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. find
  22. Yoshimasa Tsuruoka, Jun’ichi Tsujii, and Sophia Ananiadou. 2008. Accelerating the annotation of sparse named entities by dynamic sentence selection. BMC Bioinformatics, 9(Suppl 11):S8. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stenetorp-2012-brat-web-based-tool-nlp/