Elamur Research⌕
☰
home / papers / 2010 / An Overview of the CRAFT Concept Annotation Guidelines

An Overview of the CRAFT Concept Annotation Guidelines

Summary

The paper describes the concept annotation guidelines for the CRAFT corpus. It consists of 97 full-text biomedical articles (over 750 thousand words), which are annotated both syntactically and semantically. Earlier corpora were mostly limited to sentences or abstracts and to small schemas of a few dozen classes. CRAFT, by contrast, is annotated with the full term sets of eight large ontologies and terminologies, mostly from OBO. The main principles are as follows. Every mention is marked, but only of concepts that are explicitly represented in the ontology, and the selected text span must be semantically equivalent to the term. Span boundaries are determined by simple delimiter rules: whitespace, punctuation, or the beginning or end of the document, but never the middle of a word or number. Span extent is determined by syntax. Discontinuous annotations are allowed. A nested annotation is added only when its term is not a superclass of the outer one. This approach preserves precise semantics and keeps the task manageable for annotators. More than 107 thousand annotations have been created so far, and inter-annotator agreement exceeds 90% in most passes. The guidelines are not tied to a specific terminology, and others may use them under a Creative Commons license.

Key points

  • The corpus consists of 97 articles and over 750 thousand words. It includes open-access articles that curators relied on during GO annotation of laboratory mouse genes.
  • Eight resources are used for semantic annotation: GO (BP, MF, CC), Cell Type Ontology (CL), ChEBI, Sequence Ontology (SO), NCBI Taxonomy, and Entrez Gene identifiers.
  • In total, 107,836 annotations have been created. The largest numbers are for SO (32,502), GO BP (22,101), and ChEBI (15,313).
  • Inter-annotator agreement (IAA) mostly holds steadily at 90+%. The exception is the most difficult pass, GO BP & MF, where it is 80–85%. A control double-blind check on SO yielded 89.9% versus 90.4% in single-blind, so the single-blind estimate is unlikely to be substantially biased.
  • The full guidelines are published at bionlp-corpora.sourceforge.net under a Creative Commons license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bada2010overview,
  title = {{An Overview of the CRAFT Concept Annotation Guidelines}},
  author = {Bada, Michael and Eckert, Miriam and Palmer, Martha and Hunter, Lawrence},
  booktitle = {Proceedings of the Fourth Linguistic Annotation Workshop},
  year = {2010},
  url = {https://aclanthology.org/W10-1833/}
}

References

8 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, Cherry JM, Davis AP, Dolinski K, Dwight SS, Eppig JT, Harris MA, Hill DP, Issel-Tarver L, Kasarskis A, Lewis S, Matese JC, Richardson JE, Ringwald M, Rubin GM, Sherlock G. 2000. Gene Ontology: tool for the unification of biology. Nat Genetics, 25… find
  2. Bada, M. and Hunter, L. 2009a. Using Large Terminologies to Semantically Annotate Concept Mentions in Natural-Language Documents. Proceedings of the International Conference on Knowledge Capture (K-CAP) Semantic Authoring, Annotation and Knowledge Markup (SAAKM) Workshop 2009, Redondo Beach, CA, USA… find
  3. Bada, M. and Hunter, L. 2009b. Using the Gene Ontology to Annotate Biomedical Journal Articles. Proceedings of the International Conference on Biomedical Ontology (ICBO) 2009, Buffalo, NY, USA. find
  4. Bard, J., Rhee, S. Y., and Ashburner, M. 2005. An ontology for cell types. Genome Biology, 6(2), R21. find
  5. Cohen, K. B., Lanfranchi, A., Corvey, W., Baumgartner, Jr., W. A., Roeder, C., Ogren, P. V., Palmer, M., and Hunter, L. E. 2010. Annotation of all coreference in biomedical text: Guideline selection and adaptation. Proceedings of the 7 th Language Resources and Evaluation Conference (LREC) Workshop… find
  6. Degtyarenko, K., de Matos, P., Ennis, M., Hastings, J., Zbinden, M., McNaught, A., Alcántara, R., Darsow, M., Guedj, M., and Ashburner, M. 2008. ChEBI: a database and ontology for chemical entities of biological interest. Nucleic Acids Research, 36, Database Issue:D344-D350. find
  7. Eilbeck, K., Lewis, S. E., Mungall, C. J., Yandell, M., Stein, L., Durbin, R., and Ashburner, M. 2005. The Sequence Ontology: a tool for the unification of genome annotations. Genome Biology 6, R44. find
  8. Sayers, E. W., Barrett, T., Benson, D. A., Bryant, S. H., Canese, K., Chetvernin, V., Church, D. M., DiCuccio, M., Edgar, R., Federhen, S., Feolo, M., Geer, L. Y., Helmberg, W., Kapustin, Y., Landsman, D., Lipman, D. J., Madden, T. L., Maglott, D. R., Miller, V., Mizrachi, I., Ostell, J., Pruitt, K.… find

This page in Markdown → https://md.research.elamur.ai/free/papers/bada-2010-overview-craft-concept-annotation-guidelines/