📄 The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation

Summary

INCEpTION is an open-source, web-based annotation platform for semantic annotation tasks such as entity linking, fact linking, knowledge base population and semantic frame annotation. These tasks are slow and require annotators to know the underlying knowledge resource well. Building on WebAnno and UIMA, the authors combine three capabilities in a single tool: machine assistance, built-in knowledge management, and a modular, extensible architecture.

Recommenders display label suggestions inline next to existing annotations. They retrain continuously as the user accepts, rejects or adds labels. An optional active learning mode uses uncertainty sampling to direct the user to useful suggestions without restricting free annotation. RDF knowledge bases can be built and edited inside the tool or queried remotely via SPARQL, for example Wikidata or DBPedia.

As a system demonstration, the paper gives no quantitative evaluation. It reports only qualitative feedback from use cases such as FAMULUS, where annotators said an external recommender improved both speed and quality. The main contribution is cheaper bootstrapping of annotated corpora for new domains.

Key points

• The annotation schema consists of user-defined layers, each either a span or a relation between spans. Features attached to these layers may take text, numeric or true/false values, point to knowledge base concepts, or link to other annotations.

• Recommenders come in two forms:

  • Internal: implemented through a Java interface.
  • External: reached over a simple HTTP protocol that exchanges UIMA CAS XMI.

  A quality threshold can be set for each recommender so that weak models do not flood users with bad suggestions early on.

• Active learning uses uncertainty sampling and works on one layer at a time. The user can accept, reject or skip a suggestion. Skipped items come back later, and all decisions go into a learning history that can be reviewed and undone.

• Knowledge management supports classes, instances, properties and qualifiers. A configurable mapping handles Wikidata, DBPedia, OWL, CIDOC-CRM and SKOS, though advanced OWL features are deliberately not covered. A context-aware candidate ranking, adapted from Sorokin and Gurevych (2018), drives entity linking suggestions.

• The authors contrast INCEpTION with earlier tools:

  • WebAnno requires manual retraining, offers a single algorithm and shows suggestions in a split screen.
  • Prodi.gy forces users to follow the active learning order.

  Reported use cases are FAMULUS (argument components in medical case reports), EDoHa (hypothesis/evidence linking) and entity re-ranking. The software is released under Apache License 2.0.

Interesting ideas

• I1. Stage-dependent recommenders. Several recommenders run at once. Cheap, high-precision ones such as a dynamic dictionary help early on, and context-aware, recall-oriented sequence classifiers take over once more data exists. This is a practical way to bootstrap any new-domain corpus where little training data exists at first.
• I2. Advisory active learning. The system highlights the item it wants judged, but the annotator can still edit anything else, and every edit feeds back into the models. This keeps annotators productive and avoids the frustration of a rigid labeling queue.
• I3. Models as separate HTTP services. External recommenders run behind a simple HTTP interface. A crashing model cannot bring down the tool, heavy models can sit on other machines, and any programming language works. This makes it easy to plug existing pretrained models, such as Python deep learning ones, into an annotation workflow.
• I4. Event-driven modules that double as logging. The same event system that keeps recommenders updated also records user and system actions. These logs let project managers assess annotators and let researchers study whether assistance biases the labels, which is a useful evaluation habit for any assisted-annotation project.
• I5. Growing the knowledge base during annotation. Fact linking lets users mark subjects, predicates and objects in text to populate a knowledge base. This suits domains where no suitable knowledge base exists yet and one has to be built alongside the corpus.