Elamur Research⌕
☰
home / papers / 2021 / On the Contribution of Per-ICD Attention Mechanisms to Classify Health Records in Languages with Fewer Resources than English

On the Contribution of Per-ICD Attention Mechanisms to Classify Health Records in Languages with Fewer Resources than English

Paper · 2021 · Alberto Blanco, Sonja Remmer, Alicia Pérez et al.

Summary

The paper addresses automatic ICD-10 coding of clinical discharge summaries in Spanish and Swedish, two languages with far fewer clinical NLP resources than English. The authors used two hospital corpora: Basque emergency-service records covering all specialities, and Swedish gastro-surgery records from Health Bank. They retained just the 157 codes common to both corpora, every one of which falls under ICD-10 Chapter XI, the chapter covering digestive-system diseases. This choice forces the model to separate closely related diagnoses. On top of the BERT Multilingual language model, they added a classification head with a separate attention distribution over input tokens for each label. A fully connected layer then makes the multi-label prediction from these label-specific document representations. Compared with plain BERT, the head raised micro-averaged AUC substantially for Spanish and slightly for Swedish. It also roughly doubled the true positive rate in both languages. Because the per-label attention weights can be exported, the model also shows which words supported each assigned code. This makes it a plausible decision-support aid for human coders rather than a fully automatic coder.

Key points

  • Data: 3,891 Spanish EHRs and 8,909 Swedish EHRs. The Spanish vocabulary is about three times larger because the records span many specialities and include lab tests. Spanish records average 984 words (SD 491), while Swedish records average 74 words (SD 77).
  • Labels: 157 shared ICD-10 K-codes. Only 45 of these codes occur in at least 1% of records. There are 558 distinct label combinations in Spanish and 1,288 in Swedish.
  • Model: BERT Multilingual is used as a shared LM for both languages, with a per-ICD attention head that assigns a relevance to every (token, code) pair.
  • Micro-averaged AUC rose from 58.16 to 67.32 for Spanish and from 54.92 to 55.96 for Swedish. TPR rose from 17.70 to 34.92 for Spanish and from 15.49 to 27.91 for Swedish.
  • Per-class results track label frequency. True negatives barely move, true positives nearly double, and false negatives fall by about 20%. The code is available from the first author on request.

Interesting ideas

  • I1. Instead of one pooled document vector, each output label gets its own attention over the input, so every decision comes with its own evidence map over tokens
  • I2. To compare two heterogeneous corpora fairly, the authors trained and evaluated only on the label inventory both corpora share
  • I3. The task was deliberately restricted to one family of semantically close labels to stress fine-grained discrimination rather than easy coarse separation
  • I4. Gains were explained with per-class confusion matrices averaged two ways, plain and frequency-weighted, which showed where the improvement came from (more true positives, stable negatives)
  • I5. The authors profiled both corpora (document length, a spread larger than the mean, ratio of distinct label sets to records) to explain why results differ across them

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): when assigning VerbNet roles, each candidate role could keep a separate, inspectable record of which tree nodes supported it, rather than relying on one shared sentence score.
  • I2 (learn): when mixing treebanks that use different UD dialects, metrics could be reported on the intersected relation/feature set so that conversion gaps do not masquerade as parser errors.
  • I3 (world): test sets could be built from WordNet sibling senses or neighbouring verb classes, to check whether the ontology really separates near-neighbours.
  • I4 (learn): parser and tagger evaluations could report both unweighted and support-weighted per-label confusion so that rare deprels are not hidden by frequent ones.
  • I5 (tree): profiling each treebank's sentence-length spread and label-combination diversity before training would help interpret score differences between treebanks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of the Basque CountryLeioa · Spain57 works · 16,520 citations🎓Stockholm UniversityStockholm · Sweden21 works · 1,832 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Alberto Blanco, Sonja Remmer, Alicia Pérez, Hercules Dalianis and Arantza Casillas. 2021. On the Contribution of Per-ICD Attention Mechanisms to Classify Health Records in Languages with Fewer Resources than English. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{blanco2021contribution,
  title = {{On the Contribution of Per-ICD Attention Mechanisms to Classify Health Records in Languages with Fewer Resources than English}},
  author = {Alberto Blanco, Sonja Remmer, Alicia Pérez, Hercules Dalianis and Arantza Casillas},
  booktitle = {Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)},
  year = {2021},
  doi = {10.26615/978-954-452-072-4\_020},
  url = {https://doi.org/10.26615/978-954-452-072-4_020}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saadullah Amin, Günter Neumann, Katherine Dunfield, Anna Vechkaeva, Kathryn Annette Chapman, and Morgan Kelly Wixted. 2019. MLT-DFKI at CLEF eHealth 2019: Multi-label Classification of ICD-10 Codes with BERT. In CLEF (Working Notes), pages 1–15. find
  2. Alberto Blanco, Alicia Pérez, and Arantza Casillas. 2020. Extreme multi-label icd classification: Sensitivity to hospital service and time. IEEE Access, 8:183534–183545. ↗
  3. Linda Cappellato, Nicola Ferro, David E. Losada, and Henning Müller, editors. 2019. Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019, volume 2380 of CEUR Workshop Proceedings. CEUR-WS.org. find
  4. José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. 2020. Spanish Pre-Trained BERT Model and Evaluation Data. In Practical ML for Developing Countries at ICLR 2020. find
  5. Kevin Bretonnel Cohen and Dina Demner-Fushman. 2014. Biomedical natural language processing, volume 11. John Benjamins Publishing Company. find
  6. Hercules Dalianis. 2018. Clinical text mining: Secondary use of electronic patient records. Springer Nature. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv, pages 4171–4186.
  8. Jiachen Du, Lin Gui, Ruifeng Xu, and Yulan He. 2017. A convolutional attention model for text classification. In National CCF conference on natural language processing and Chinese computing, pages 183–195. Springer. find
  9. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pretraining for biomedical natural language processing. arXiv preprint arXiv:2007.15779. ↗
  10. Aron Henriksson, Martin Hassel, and Maria Kvist. 2011. Diagnosis code assignment support using random indexing of patient records – a qualitative feasibility study. In Conference on Artificial Intelligence in Medicine in Europe, pages 348–352. Springer. find
  11. Rita Kukafka, Michael E Bales, Ann Burkhardt, and Carol Friedman. 2006. Human and automated coding of rehabilitation discharge summaries according to the International Classification of Functioning, Disability, and Health. Journal of the American Medical Informatics Association, 13(5):508–515. ↗
  12. Guillermo López-Garcıa, José M Jerez, and Francisco J Veredas. 2020. ICB-UMA at CLEF e-Health 2020 Task 1: Automatic ICD-10 coding in Spanish with BERT. In Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020. find
  13. Martin Malmsten, Love Börjeson, and Chris Haffenden. 2020. Playing with Words at the National Library of Sweden – Making a Swedish BERT. arXiv preprint arXiv:2007.01658. ↗
  14. Antonio Miranda-Escalada, Aitor Gonzalez-Agirre, Jordi Armengol-Estapé, and Martin Krallinger. 2020. Overview of automatic clinical coding: annotations, guidelines, and solutions for non-english clinical cases at codiesp track of CLEF eHealth 2020. In Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings. find
  15. Aurélie Névéol, Hercules Dalianis, Sumithra Velupillai, Guergana Savova, and Pierre Zweigenbaum. 2018a. Clinical natural language processing in languages other than English: opportunities and challenges. Journal of biomedical semantics, 9(1):1–13. find
  16. Aurélie Névéol, Aude Robert, Francesco Grippo, Claire Morgand, Chiara Orsi, Laszlo Pelikan, Lionel Ramadier, Grégoire Rey, and Pierre Zweigenbaum. 2018b. CLEF eHealth 2018 Multilingual Information Extraction Task Overview: ICD10 Coding of Death Certificates in French, Hungarian and Italian. In CLEF (Working Notes), pages 1–18. find
  17. Maite Oronoz, Koldo Gojenola, Alicia Pérez, Arantza Dı́az de Ilarraza, and Arantza Casillas. 2015. On the creation of a clinical gold standard corpus in Spanish: Mining adverse drug reactions. Journal of biomedical informatics, 56:318–332. ↗
  18. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. ↗
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  20. Sonja Remmer, Anastastios Lamproudis, and Hercules Dalianis. 2021. Multi-label Diagnosis Classification of Swedish Discharge Summaries – ICD-10 Code Assignment Using KB-BERT. In the Proceedings of Recent Advances in Natural Language Processing, RANLP 2021, Varna, Bulgaria. find
  21. Anders Jacobsson and Lisbeth Serdén. 2013. ningskvalitet i patientregistret, (In Swedish). find
  22. WHO. 2016. International Classification of Diseases (ICD). Accessed 2021-04-14. find
  23. Kod-Zachariah Zhang, Jingshu Liu, and Narges Razavian. 2020. BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining. arXiv preprint arXiv:2006.03685. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/blanco-2021-contribution-per-icd-attention-mechanisms/