Elamur Research⌕
☰
home / papers / 2017 / NegBio: a high-performance tool for negation and uncertainty detection in radiology reports

NegBio: a high-performance tool for negation and uncertainty detection in radiology reports

Paper · 2017 · Yifan Peng, Xiaosong Wang, Le Lu et al.

Summary

Radiology reports often mention findings only to rule them out ("no pneumothorax") or to hedge about them ("suspicious pneumothorax"). Extraction systems that ignore this return many false hits. NegBio is an open-source, rule-based tool that labels each pre-recognized finding mention as negated, uncertain, or neither. Instead of NegEx-style regular expressions over surface word windows, it writes its rules as Semgrex patterns over a universal dependency graph. Sentences are parsed with the Bllip biomedical model and converted with the Stanford converter (CCProcessed, Universal). A trigger's scope then follows syntax rather than word distance.

Findings come from MetaMap, restricted to 14 chest finding types. Evaluation used OpenI, a new 900-report ChestX-ray corpus, BioScope and the general clinical PK corpus. NegBio beat NegEx on every dataset, by an average of 9.5% in precision and 5.1% in F1. This shows that syntax-aware rules can be both stricter and broader than lexical ones while staying transparent and requiring no training data.

Key points

  • Datasets:
    • OpenI: 3,851 reports with 1,354 positive findings.
    • ChestX-ray: 900 reports with 2,131 positive findings, Cohen's kappa 84.3%.
    • BioScope test set: 977 items with 466 negations.
    • PK: 116 items with 491 negated phrases.
  • Patterns were designed on 80% of OpenI and tested on the other 20%. ChestX-ray was used entirely as an untuned test set.
  • Adding any negation/uncertainty filter after MetaMap raised end-to-end precision on findings from 13.8% to 77.2%.
  • BioScope results:
    • NegBio: P 96.1, R 95.7, F 95.9.
    • NegEx: P 70.6, R 98.7, F 82.3.
  • PK results:
    • NegBio: F 93.3.
    • NegEx: F 93.1.
  • Turning off the uncertainty rules lowered F-score by 7.4% on OpenI and 2.5% on ChestX-ray (relative). Remaining errors fall into three groups:
    • MetaMap NER misses, e.g. about 30% of Nodule mentions in OpenI.
    • Misparsed verbless noun-phrase sentences.
    • Unhandled double negation.

Interesting ideas

  • I1. Negation scope is defined by the matched dependency subgraph rather than by a token window, so coordinated items far from the trigger are still covered and unrelated clauses are excluded.
  • I2. Uncertainty is treated as a separate label next to negation, and the ablation shows it matters for precision.
  • I3. Rules include sibling patterns, such as "not" attached to "excluded" that scopes over the excluded noun, as well as simple chains.
  • I4. Patterns were written on one 80% split and then applied unchanged to a second, independently annotated corpus to test robustness.
  • I5. The error analysis traces failures back to the earlier pipeline stages, both recognition and parsing of verbless fragments, not just the rules.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Polarity and scope of "not/no/without" could be assigned to frame arguments by graph patterns over the UD tree, making each negated role traceable to the edge that licensed it.
  • I2 (world): Entity states could carry a three-way status (asserted, denied, hedged), so hedged facts like "suspected X" are stored without being treated as true when answering.
  • I3 (frames): Rule matching on the tree should support sibling and subgraph shapes, which covers verbs whose meaning flips under negation (exclude, rule out, deny).
  • I4 (learn): Hand-written or self-induced repair rules should be checked on a corpus never consulted during rule design, as a guard against overfitting rules to one dialect.
  • I5 (tree): Telegraphic noun-phrase sentences ("No definite effusion seen, no edema") deserve a dedicated treebank slice or conversion check, since downstream rules silently fail when the parser mishandles them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️National Institutes of HealthBethesda · United States9 works · 3,833 citations🔬National Center for Biotechnology InformationBethesda · United States3 works · 1,457 citations

Also on the byline: National Institutes of Health Clinical Center.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Biochemistry, Genetics and Molecular Biology · 116 works Molecular Biology · 114 works

Cite

Cite as Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers and Zhiyong Lu. 2017. NegBio: a high-performance tool for negation and uncertainty detection in radiology reports. arXiv:1712.05898.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1712.05898 · arXiv 1712.05898

@misc{peng2017negbio,
  title = {{NegBio: a high-performance tool for negation and uncertainty detection in radiology reports}},
  author = {Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers and Zhiyong Lu},
  year = {2017},
  doi = {10.48550/arxiv.1712.05898},
  eprint = {1712.05898},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1712.05898}
}

References

34 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wendy W Chapman, Will Bridewell, Paul Hanbury, Gregory F Cooper, and Bruce G Buchanan. Evaluation of negation phrases in narrative clinical reports. In Proceedings of the AMIA Symposium, pages 105–109, 2001. find
  2. Stephen Wu, Timothy Miller, James Masanz, Matt Coarr, Scott Halgrim, David Carrell, and Cheryl Clark. Negations not solved: generalizability versus optimizability in clinical natural language processing. PLoS ONE, 9(11):e112774, 2014. ↗
  3. George Gkotsis, Sumithra Velupillai, Anika Oellrich, Harry Dean, Maria Liakata, and Rina Dutta. Dont let notes be misunderstood: a negation detection method for assessing risk of suicide in mental health records. In Proceedings of the 3rd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 95–105, 2016. find
  4. Roser Morante. Descriptive analysis of negation cues in biomedical texts. In International Conference on Language Resources and Evaluation (LREC), pages 1429–1436, 2010. find
  5. Andrew S. Wu, Bao H. Do, Jinsuh Kim, and Daniel L. Rubin. Evaluation of negation and uncertainty detection and its impact on precision and recall in search. Journal of Digital Imaging, 24(2):234–242, 2011. find
  6. C Friedman, G Hripcsak, L Shagina, and H Liu. Representing information in patient reports using natural language processing and the extensible markup language. Journal of the American Medical Informatics Association : JAMIA, 6:76–87, 1999. ↗
  7. Wendy W Chapman, Will Bridewell, Paul Hanbury, Gregory F Cooper, and Bruce G Buchanan. A simple algorithm for identifying negated findings and diseases in discharge summaries. Journal of Biomedical Informatics, 34(5):301–310, 2001. ↗
  8. Henk Harkema, John N Dowling, Tyler Thornblade, and Wendy W Chapman. ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports. Journal of biomedical informatics, 42(5):839–851, 2009. ↗
  9. Brian E Chapman, Sean Lee, Hyunseok Peter Kang, and Wendy W Chapman. Document-level classification of ct pulmonary angiography reports based on an extension of the context algorithm. Journal of biomedical informatics, 44(5):728–737, 2011. ↗
  10. Alan R Aronson and François-Michel Lang. An overview of MetaMap: historical perspective and recent advances. Journal of the American Medical Informatics Association, 17(3):229–236, 2010. ↗
  11. Pradeep G Mutalik, Aniruddha Deshpande, and Prakash M Nadkarni. Use of general-purpose negation detection to augment concept indexing of medical documents: a quantitative study using the UMLS. Journal of the American Medical Informatics Association, 8(6):598–609, 2001. ↗
  12. Sunghwan Sohn, Stephen Wu, and Christopher G Chute. Dependency parser-based negation detection in clinical narratives. In AMIA Summits on Translational Science Proceedings, volume 2012, pages 1–8, 2012. find
  13. Saeed Mehrabi, Anand Krishnan, Sunghwan Sohn, Alexandra M Roch, Heidi Schmidt, Joe Kesterson, Chris Beesley, Paul Dexter, C Max Schmidt, Hongfang Liu, and Mathew Palakal. DEEPEN: A negation detection system for clinical text incorporating dependency relation into NegEx. Journal of Biomedical Informatics, 54:213– 219, 2015. ↗
  14. Yang Huang and Henry J Lowe. A novel hybrid approach to automated negation detection in clinical radiology reports. Journal of the American Medical Informatics Association, 14(3):304–311, 2007. ↗
  15. Cheryl Clark, John Aberdeen, Matt Coarr, David Tresner-Kirsch, Ben Wellner, Alexander Yeh, and Lynette Hirschman. MITRE system for clinical assertion status classification. Journal of the American Medical Informatics Association, 18(5):563–567, 2011. ↗
  16. Sergey Goryachev, Margarita Sordo, Qing T Zeng, and Long Ngo. Implementation and evaluation of four different methods of negation detection. Technical report, DSG, Brigham and Womens Hospital, Harvard Medical School, 2006. find
  17. Philip V. Ogren, Guergana K. Savova, and Christopher G. Chute. Constructing evaluation corpora for automated clinical named entity recognition. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC), pages 28–30, 2008. find
  18. Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552–556, 2011. find
  19. Hanna Suominen, Sanna Salanterä, Sumithra Velupillai, Wendy W Chapman, Guergana Savova, Noemie Elhadad, Sameer Pradhan, Brett R South, Danielle L Mowery, Gareth JF Jones, et al. Overview of the ShARe/CLEF eHealth evaluation lab 2013. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 212–231, 2013. find
  20. Daniel Albright, Arrick Lanfranchi, Anwen Fredriksen, William F Styler, Colin Warner, Jena D Hwang, Jinho D Choi, Dmitriy Dligach, Rodney D Nielsen, James Martin, et al. Towards comprehensive syntactic and semantic annotations of the clinical narrative. Journal of the American Medical Informatics Association, 20(5):922–930, 2013. find
  21. Marie-Catherine De Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D Manning. Universal Stanford dependencies: A cross-linguistic typology. In Proceedings of 9th International Conference on Language Resources and Evaluation (LREC), volume 14, pages 4585–4592, 2014. our card
  22. Haibin Liu, Lawrence Hunter, Vlado Kešelj, and Karin Verspoor. Approximate subgraph matching-based literature mining for biomedical events and relations. PLoS ONE, 8(4):e60954, 2013. ↗
  23. Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–310, 2015. ↗
  24. Veronika Vincze. Speculation and negation annotation in natural language texts: what the case of BioScope might (not) reveal. In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing (NeSp-NLP), pages 28–31, 2010. our card
  25. Dina Demner-Fushman, Willie J Rogers, and Alan R Aronson. MetaMap Lite: an evaluation of a new java implementation of metamap. Journal of the American Medical Informatics Association, pages 1–5, 2017. ↗
  26. Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media, Inc., 2009. find
  27. Eugene Charniak and Mark Johnson. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL), pages 173–180, 2005. our card
  28. David McClosky. Any domain parsing: automatic domain adaptation for natural language parsing. Thesis, Department of Computer Science, Brown University, 2009. ↗
  29. Marie-Catherine De Marneffe and Christopher D Manning. Stanford typed dependencies manual, 2015. find
  30. Nathanael Chambers, Daniel Cer, Trond Grenager, David Hall, Chloe Kiddon, Bill MacCartney, Marie-Catherine De Marneffe, Daniel Ramage, Eric Yeh, and Christopher D Manning. Learning alignments and leveraging natural logic. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 165–170, 2007. find
  31. Xiaosong Wang, Yifan Peng, Le Lu, Mohammadhadi Bagheri, Zhiyong Lu, and Ronald Summers. ChestX-ray8: hospital-scale Chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. find
  32. Mary L McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282, 2012. ↗
  33. Houssam Nassif, Ryan Woods, Elizabeth Burnside, Mehmet Ayvaci, Jude Shavlik, and David Page. Information extraction for clinical data mining: a mammography case study. In IEEE International Conference on Data Mining, pages 37–42, 2009. find
  34. Wendy W Chapman, Dieter Hilert, Sumithra Velupillai, Maria Kvist, Maria Skeppstedt, Brian E Chapman, Michael Conway, Melissa Tharp, Danielle L Mowery, and Louise Deleger. Extending the negex lexicon for multiple languages. Studies in health technology and informatics, 192:677–681, 2013. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2017-negbio-high-performance-tool-negation/