Elamur Research⌕
☰
home / papers / 2026 / AnnoHID: LLM-Assisted Annotation Framework for Low-Resource Medical Texts

AnnoHID: LLM-Assisted Annotation Framework for Low-Resource Medical Texts

Paper · 2026 · Annisa Maulida Ningtyas, Guntur Budi Herwanto, Yunita Sari et al.

Summary

Training Medical Concept Normalization (MCN) systems requires annotated data that links everyday patient wording to standard clinical vocabularies. Such data is costly to build and practically absent for languages like Bahasa Indonesia. Existing tools (BRAT, INCEpTION, MedCAT, cTAKES, SciSpaCy and others) mostly target English. AnnoHID is an open-source, web-based, human-in-the-loop platform for these settings. Gemini-2.5 Flash proposes NER spans and SNOMED-CT links, and annotators accept, reject or correct each suggestion. No Indonesian edition of SNOMED-CT exists yet, so the model works in three steps: it translates each entity into English, rewrites it as clinical terminology, and then maps it to a concept ID through the SNOMED-CT API. A user study with 21 participants compared assisted and manual annotation on Indonesian health Q&A posts. Assistance produced better inter-annotator agreement, and human review improved the raw LLM output. However, it did not save time and appeared to make annotators over-trust suggested concepts. The framework offers a reusable route to medical datasets for under-resourced languages.

Key points

  • Data and participants: The study used 23 posts from Alodokter about diabetes mellitus, tuberculosis and stroke. Participants were one expert medical coder and 20 health-information students, split into a manual group and an LLM-assisted group. The expert worked in the manual group and served as the reference.
  • Agreement and LLM baseline: The LLM-assisted setting reached κ = 0.76 for NER and κ = 0.63 for MCN. Raw Gemini strict NER F1 against the expert was 0.39, rising to 0.45 after human review.
  • MCN results: Span F1 went from 0.39 to 0.46 after review, while concept accuracy dipped slightly from 0.55 to 0.54. The authors attribute this to normalization bias, where annotators accept suggested SNOMED-CT IDs without checking them.
  • Annotation time: Assisted annotators were slightly slower per post, at 133.2s vs 117.8s for NER and 275.1s vs 262.8s for MCN.
  • Availability: The system uses LangExtract for pre-annotation and is released under the MIT License on GitHub (gunturbudi/annohid), with a public demo.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓TU WienVienna · Austria7 works · 786 citations🎓Universitas Gadjah MadaYogyakarta · Indonesia4 works · 56 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 1Research subjects and languages the work is about.Low-resource languages · 267 works

Cite

Cite as Annisa Maulida Ningtyas, Guntur Budi Herwanto, Yunita Sari, Rifki Afina Putri, Filip Kovacevic, Alaa El-Ebshihy, Varvara Arzt and Florina Piroi. 2026. AnnoHID: LLM-Assisted Annotation Framework for Low-Resource Medical Texts. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-demo.67

@inproceedings{ningtyas2026annohid,
  title = {{AnnoHID: LLM-Assisted Annotation Framework for Low-Resource Medical Texts}},
  author = {Annisa Maulida Ningtyas, Guntur Budi Herwanto, Yunita Sari, Rifki Afina Putri, Filip Kovacevic, Alaa El-Ebshihy, Varvara Arzt and Florina Piroi},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.67},
  url = {https://doi.org/10.18653/v1/2026.acl-demo.67}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asma Ben Abacha and Pierre Zweigenbaum. 2011. Medical Entity Recognition: A Comparaison of Semantic and Statistical Methods. In Proceedings of BioNLP 2011 Workshop, pages 56–64, Portland, Oregon, USA. Association for Computational Linguistics. find
  2. Alexander S Choi, Syeda Sabrina Akter, JP Singh, and Antonios Anastasopoulos. 2024. The llm effect: Are humans truly using llms, or are they being influenced by them instead? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22032–22054. find
  3. Leonard W D’Avolio, Thien M Nguyen, Sergey Goryachev, and Louis D Fiore. 2011. Automated concept-level information extraction to reduce the need for custom software and rules development. Journal of the American Medical Informatics Association, 18(5):607–613. find
  4. Guy Divita, Qing T Zeng, Adi V. Gundlapalli, Scott Duvall, Jonathan Nebeker, and Matthew H. Samore. 2014. Sophia: A Expedient UMLS Concept Extraction Annotator. AMIA Annual Symposium Proceedings, 2014:467–476. Sophia. find
  5. Dmitriy Dligach, Steven Bethard, Lee Becker, Timothy Miller, and Guergana K Savova. 2014. Discovering body site and severity modifiers in clinical texts. Journal of the American Medical Informatics Association, 21(3):448–454. find
  6. A. M. Fage-Butler and M. Nisbeth Jensen. 2016. Medical terminology in online patient–patient communication: evidence of high health literacy? Health Expectations, 19(3):643–653. find
  7. Genevieve Gorrell, Xingyi Song, and Angus Roberts. 2018. Bio-YODIE: A Named Entity Linking System for Biomedical Text. arXiv preprint. ArXiv:1811.04860 [cs]. — [arXiv:1811.04860](https://arxiv.org/abs/1811.04860) find
  8. Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pa…
  9. Zeljko Kraljevic, Thomas Searle, Anthony Shek, Lukasz Roguski, Kawsar Noor, Daniel Bean, Aurelie Mascio, Leilei Zhu, Amos A Folarin, Angus Roberts, and 1 others. 2021. Multi-domain clinical natural language processing with medcat: the medical concept annotation toolkit. Artificial intelligence in me… find
  10. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. CoRR, abs/1603.01360.
  11. Fei Li, Weisong Liu, and Hong Yu. 2018. Extraction of Information Related to Adverse Drug Events from Electronic Health Record Notes: Design of an End-to-End Model Based on Deep Learning. JMIR Medical Informatics, 6(4):e12159. Company: JMIR Medical Informatics Distributor: JMIR Medical Informatics I… find
  12. Z. Miftahutdinov and E. Tutubalina. 2019. Deep neural models for medical concept normalization in user-generated texts. In Proc. of the 57th Annual Meeting of the ACL: Student Research Workshop, pages 393– 399, Florence, Italy. ACL. find
  13. Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319–327. ArXiv:1902.07669 [cs].
  14. Annisa Maulida Ningtyas, Alaa El-Ebshihy, Florina Piroi, and Allan Hanbury. 2024. Improving laypeople familiarity with medical terms by informal medical entity linking. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 113–126, Cham. Springer Nature Switzerland. find
  15. Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute. 2010. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. Journal of the American Medical Informati… find
  16. Hope Schroeder, Deb Roy, and Jad Kabbara. 2025. Just put a human in the loop? investigating LLM-assisted annotation for subjective tasks. In Findings of the Association for Computational Linguistics: ACL 2025, pages 25771–25795, Vienna, Austria. Association for Computational Linguistics. find
  17. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. brat: a Web-based Tool for NLP-Assisted Text Annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, page…
  18. Xiaoyan Wang, A.D Dave, G Ruaño, and J Kost. 2022. Automated Extraction of Pain Symptoms: A Natural Language Approach using Electronic Health Records. Pain Physician. find
  19. Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. Human-llm collaborative annotation through effective verification of llm labels. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Mach… find
  20. Honghan Wu, Giulia Toti, Katherine I Morley, Zina M Ibrahim, Amos Folarin, Richard Jackson, Ismail Kartoglu, Asha Agrawal, Clive Stringer, Darren Gale, Genevieve Gorrell, Angus Roberts, Matthew Broadbent, Robert Stewart, and Richard Jb Dobson. 2018. SemEHR: A general-purpose semantic search system t… find
  21. Yuhao Zhang, Yuhui Zhang, Peng Qi, Christopher D Manning, and Curtis P Langlotz. 2021. Biomedical and clinical English model packages for the Stanza Python NLP library. Journal of the American Medical Informatics Association, 28(9):1892–1899. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ningtyas-2026-annohid-llm-assisted-annotation-framework/