Elamur Research⌕
☰
home / papers / 2026 / BNLP: A Text Annotation Platform for Quality Control of LLM-Generated Annotations

BNLP: A Text Annotation Platform for Quality Control of LLM-Generated Annotations

Paper · 2026 · Xinhao Zhuang, Qiongyu Tian, Yalin Chen et al.

Summary

Building annotated corpora by hand is slow and expensive, especially when resources are scarce. Existing annotation software typically offers no LLM integration at all, or applies LLMs just once to pre-label data without any built-in review or quality checks. The authors present BNLP, an open-source annotation platform in which LLM output is a draft that humans must accept, edit or reject. Prompts are built from gold-standard data, with few-shot examples chosen by dense retrieval. Three roles (Project Manager, Annotator, Auditor) work through repeated review rounds, while dashboards track inter-annotator agreement. The platform supports entities, attributes and relations, nested and discontinuous spans, linking to UMLS, and import and export in Excel and JSON. In a P450 enzyme-reaction case study measured against P450RDB, LLM pre-annotation followed by human revision cut annotation time by 74.3% and raised F1 by 11.6% compared with fully manual work. This suggests that LLM speed can be used without giving up data reliability. The platform does not depend on any particular domain, although it has so far been tested only on biomedical data.

Key points

  • Quality-control design:
    • Agreement is measured with Fleiss' Kappa and Krippendorff's Alpha and shown as heatmaps.
    • LLM pre-annotations are automatically compared with human revisions.
    • Auditors can send a task back to annotators with a "Relabel" request.
  • LLM setup:
    • Calls go through the OpenAI-compatible API.
    • Experiments used DeepSeek-V3 with temperature 0.1.
    • Long documents are split to fit the context window, and results are merged at the document level.
  • P450 study:
    • Corpus: 12 PubMed articles, with efficiency tested on 20 segments (3,914 words).
    • LLM-only pre-annotation was 95.4% faster than manual work but less accurate.
    • LLM plus human revision was 74.3% faster and gained +11.6% F1.
  • FMH case study: The project produced 19,536 structured knowledge fragments for a knowledge graph.
  • Limitations and availability:
    • Evaluation covers only biomedical data (BC5CDR, P450).
    • The annotation–training–evaluation loop runs offline and is not yet part of the interactive workflow.
    • Code is available at GitHub BioMedBigDataCenter/BNLP.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🏛️Chinese Academy of SciencesBeijing · China246 works · 24,951 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,743 citations

Also on the byline: Shanghai Institute of Nutrition and Health.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksGeneral Social Sciences · 254 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Xinhao Zhuang, Qiongyu Tian, Yalin Chen, Tianle Xin, Yongyong Fu, Yunchao Ling and Guoqing Zhang. 2026. BNLP: A Text Annotation Platform for Quality Control of LLM-Generated Annotations. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1185

@inproceedings{zhuang2026bnlp,
  title = {{BNLP: A Text Annotation Platform for Quality Control of LLM-Generated Annotations}},
  author = {Xinhao Zhuang, Qiongyu Tian, Yalin Chen, Tianle Xin, Yongyong Fu, Yunchao Ling and Guoqing Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1185},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1185}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. AnnoLLM: Making large language models to be better crowdsourced annotators. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com…
  2. Ornella Irrera, Stefano Marchesin, Farzad Shami, and Gianmaria Silvello. 2025. Doctron: A web-based collaborative annotation tool for ground truth creation in ir. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3488–3497. find
  3. Rezarta Islamaj, Dongseop Kwon, Sun Kim, and Zhiyong Lu. 2020. Teamtat: a collaborative text annotation tool. Nucleic acids research, 48(W1):W5– W11. find
  4. Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang. 2024. MEGAnno+: A human-LLM collaborative annotation system. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 168–176, St. Julians, Ma… find
  5. Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. INCePTiON: A high-level platform for iterative corpus annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 5–9, Santa Fe, New… find
  6. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. find
  7. Ines Montani and Matthew Honnibal. 2018. Prodigy: A modern annotation tool for AI, Machine Learning and NLP. Explosion AI, Berlin, Germany. find
  8. Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text annotation tool for human. Software available from https://github.com/doccano/doccano. ↗
  9. Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. 2022. POTATO: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 327–33…
  10. Jie Yang, Yue Zhang, Linwei Li, and Xingxuan Li. 2018. YEDDA: A lightweight collaborative text annotation tool. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (System Demonstrations), pages 31–36, Melbourne, Australia. Association for Computational Linguis… find
  11. Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. 2025. Data-centric artificial intelligence: A survey. ACM Computing Surveys, 57(5):1–42. find
  12. Yang Zhang, Xianrun Pan, Tianyu Shi, Zhifeng Gu, Zhaochang Yang, Minghao Liu, Yi Xu, Yu Yang, Liping Ren, Xiaoming Song, and 1 others. 2024. P450rdb: A manually curated database of reactions catalyzed by cytochrome p450 enzymes. Journal of advanced research, 63:35–42. find
  13. Tal Perry. 2021. Lighttag: Text annotation platform. In Proceedings of the 2021 conference on empirical methods in natural language processing: system demonstrations, pages 20–27. find
  14. Jiahe Song, Hongxin Ding, Zhiyuan Wang, Yongxin Xu, Yasha Wang, and Junfeng Zhao. 2024. ITAKE: Interactive unstructured text annotation and knowledge extraction system with LLMs and ModelOps. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System… find
  15. Axel Sorensen, Siyao Peng, Barbara Plank, and Rob Van Der Goot. 2024. Eevee: An easy annotation tool for natural language processing. In Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 216–221. find
  16. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. brat: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, page…
  17. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang, and Chen Guo. 2025. GPT-NER: Named entity recognition via large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4257–4275, Albuquerque, New Mexico. Assoc…
  18. Xi Xu, Sumin Jo, Adam Officer, Angela Chen, Yufei Huang, and Lei Li. 2025. BioGraphia: A LLM-assisted biological pathway graph annotation platform. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 480–486, Suzhou, China. Associat… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhuang-2026-bnlp-text-annotation-platform-quality/