📄 Detection of Adverse Drug Events in Dutch clinical free text documents using Transformer Models: benchmark study

Summary

The paper builds a first reference point for automatically finding adverse drug events (ADEs) in Dutch hospital texts. Most clinical NLP work targets English, and earlier ADE studies often used metrics that suit rare events poorly and lacked external testing. The authors train a Bi-LSTM baseline and four encoder models (BERTje, RobBERT, MedRoBERTa.nl, NuNER) on the Dutch ADE corpus. This corpus has 102 densely annotated ICU progress notes about drug-related acute kidney injury. The pipeline first tags drugs and disorders with NER. A relation classifier then decides which drug–disorder pairs form an ADE, running either on gold entities or on predicted ones. The models are also tested on a second ICU corpus and, for external validation, on discharge letters from two other hospitals (WINGS). MedRoBERTa.nl comes out on top, though the gaps between models are small. The study argues that ADE systems should be judged with imbalance-aware measures and with clinically realistic scenarios in mind.

Key points

• Models compared: Bi-LSTM, BERTje, RobBERT, MedRoBERTa.nl, NuNER. Generative decoder-only models were left out on purpose.
• Training data is the Dutch ADE corpus: 102 ICU notes with just over 600 ADE-related drug–disorder pairs. Every note contains at least one drug-related AKI.
• MedRoBERTa.nl reached a macro-averaged F1 of 0.63 on the relation task with gold entities and 0.62 with predicted entities.
• On WINGS (100 discharge letters, 36 with expert-confirmed ADEs), the MedRoBERTa.nl models flagged 67–74% of the letters that contain ADEs.
• Reported metrics include micro- and macro-F1. F1 and F2 were used for threshold selection, since recall matters most, and the authors recommend precision-recall curves.

Interesting ideas

• I1. An ADE is modelled as a relation between a drug mention and a disorder mention, not as a single entity label; the authors say this avoids preventable classification errors.
• I2. Each mention-level relation is mapped to a unique ADE group, so repeated mentions and several suspected drugs for one disorder count as one clinical event.
• I3. The relation step is scored twice, once on gold entities (two-step) and once on predicted entities (end-to-end), to separate the classifier's own errors from errors carried over from NER.
• I4. The relation classifier reuses the NER model's entity embeddings instead of fine-tuning a separate transformer, which makes relation classification cheaper.
• I5. The training corpus has an ADE in every note, so the authors validate on corpora that include ADE-free documents and choose thresholds with recall-weighted F2.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): encode causal clinical or other cause–effect claims as a frame with a causer and an affected role rather than a tagged span, so the explanation shows which two participants are linked.
• I2 (world): when building the world of a text, merge repeated event mentions into one event node and allow several candidate causes, so counts reflect events and not sentences.
• I3 (learn): evaluate each stage with gold input and with predicted upstream input, so error propagation from tokens/tree into frames can be measured directly.
• I4 (tree): gparse's contextual token vectors could feed a light pair classifier for roles or arcs, avoiding a second heavy encoder.
• I5 (learn): include negative-control texts with no target phenomenon and report macro scores, so rare-class detectors are not rewarded for always firing.