📄 SINAI at #SMM4H–HeaRD 2026: Multilingual Clinical NER with MrBERT-biomed and Optuna Hyperparameter Optimization

Summary

The SINAI team (Universidad de Jaén) developed an entry for MultiClinAI, one of the shared tasks hosted at SMM4H-HeaRD 2026. The task is clinical NER for DISEASE, PROCEDURE and SYMPTOM mentions, scored on Spanish texts with strict span-level micro-F1. The MultiClinNER corpus provides annotated clinical documents in seven European languages. The team used all seven for training and kept Spanish for validation and testing.

The system has three main parts:

• Separate models per entity type. They fine-tuned three BIO taggers from MrBERT-biomed, a ModernBERT-based encoder that BSC-LT further pre-trained on biomedical literature.
• Automatic tuning. Optuna selected the hyperparameters for each model.
• Boundary correction. A rule-based step adjusts predicted spans to full words, so that subword tokenization does not cost points under strict matching.

In development, biomedical pre-training and multilingual data each helped, and they worked best together. On the official test set the system reached F1 of 0.745 (DISEASE), 0.711 (PROCEDURE) and 0.660 (SYMPTOM).

Recall dropped sharply on natively written clinical notes compared with translated ones. This suggests that real clinical writing style is a key remaining obstacle.

Key points

• Encoder: MrBERT-biomed was created from MrBERT-es by continued pre-training on about 24B biomedical tokens, mostly English (84.7%) and Spanish (14.8%). Its context window is 8,192 tokens.
• Hyperparameter search: Optuna ran 20 trials per entity model with a TPE sampler and a median pruner. The search covered learning rate, weight decay, warmup, batch size, epochs and gradient accumulation. The selected learning rates ranged from 2.4e-4 to 4.9e-4.
• Development results: On the Spanish validation set, MrBERT-biomed trained on multilingual data scored best for DISEASE (0.7478) and SYMPTOM (0.7345). For PROCEDURE it scored 0.7597, slightly below MrBERT-es (0.7625). Training on Spanish only was the weakest setup.
• Native-text gap: On the test set's Native subcorpus, recall fell to 0.501 for PROCEDURE and 0.505 for SYMPTOM. The authors attribute this to abbreviations, idiomatic phrasing and jargon.
• Training and post-processing details: Training used HuggingFace Transformers on a single 40 GB Ampere GPU with BF16, 512-token windows and a stride of 128. The word-boundary rule is built for Spanish orthography and would need changes for other languages.