Elamur Research⌕
☰
home / papers / 2019 / Inferring Which Medical Treatments Work from Reports of Clinical Trials

Inferring Which Medical Treatments Work from Reports of Clinical Trials

Paper · 2019 · Eric Lehman, Jay DeYoung, Regina Barzilay et al.

Summary

Clinical trial results are mostly published as prose. With more than a hundred RCT reports appearing each day, evidence-synthesis groups such as Cochrane cannot keep up. The authors define a task called "evidence inference." A system receives a full-text RCT article and a prompt naming an intervention, a comparator and an outcome. It must decide whether the intervention produced a significant increase, a significant decrease, or no significant difference, and point to the supporting text.

To support the task, they built a corpus of over 10,000 prompts linked to full articles. Medical doctors recruited via Upwork created the labels and evidence spans, a second group annotated them independently, and a third group verified them. Baselines range from rule-based heuristics to neural models, including attention conditioned on the prompt and pretrained on marked rationales.

Neural models score highest, but attention adds only modest gains. A model given just the gold evidence snippets outperforms the best end-to-end system by more than 20 F1 points. Finding the relevant passage in a long document is therefore the central difficulty. The released corpus, leaderboard and code provide a benchmark for this problem.

Key points

  • Prompts are short (about 13.5 words), while articles are long (about 4,200 words). Only 40.5% of evidence spans fall in the abstract, so full texts are required.
  • Of 3,525 candidate articles, 1,106 (31.4%) were discarded as invalid, mostly because they were not RCTs. Valid articles average 4.19 prompts each.
  • In 67% of articles, at least two prompts carry different labels. Generic positive wording can therefore mislead a model, acting like built-in adversarial examples.
  • Annotator agreement (Krippendorff's α) rose from 0.58 to 0.84 after targeted feedback. On 113 new prompts it was 0.75 among annotators and 0.80 with the prompt generators. A total of 16 doctors took part.
  • Dropping the article and using only the prompt lowers F1 by just ~5 points. Even pretrained attention places only about 10–15% of its mass on true evidence tokens and appears to "forget" this supervision during fine-tuning. Resources are available at evidence-inference.ebm-nlp.com.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States159 works · 65,157 citations🎓Northeastern UniversityBoston · United States28 works · 6,164 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Biochemistry, Genetics and Molecular Biology · 116 works Molecular Biology · 114 works

Cite

Cite as Eric Lehman, Jay DeYoung, Regina Barzilay and Byron C. Wallace. 2019. Inferring Which Medical Treatments Work from Reports of Clinical Trials. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1371

@inproceedings{lehman2019inferring,
  title = {{Inferring Which Medical Treatments Work from Reports of Clinical Trials}},
  author = {Eric Lehman, Jay DeYoung, Regina Barzilay and Byron C. Wallace},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1371},
  url = {https://doi.org/10.18653/v1/n19-1371}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473. arXiv preprint Hilda Bastian, Paul Glasziou, and Iain Chalmers. 2010. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? P… our card
  2. Catherine Blake and Ana Lucic. 2015. Automatic endpoint detection to support the systematic review process. Journal of biomedical informatics, 56:42–56. find
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 632–642. ACL. find
  4. Claire Cardie. 1997. Empirical methods in information extraction. AI Magazine, 18(4):65. find
  5. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  6. Eunsol Choi, Daniel Hewlett, Alexandre Lacoste, Illia Polosukhin, Jakob Uszkoreit, and Jonathan Berant. 2017. Coarse-to-fine question answering for long documents. In Proceedings of the Association for Computational Linguistics. ACL. find
  7. Sanda M Harabagiu, Marius A Paşca, and Steven J Maiorano. 2000. Experiments with open-domain textual question answering. In Proceedings of the Association for Computational Linguistics, pages 292–298. ACL. find
  8. Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Michael Junk, and Chin-Yew Lin. 2000. Question answering in webclopedia. In TREC, volume 52, pages 53–56. find
  9. Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 12–22. Association for Computational Linguistics. find
  10. Divyansh Kaushik and Zachary C Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. arXiv preprint arXiv:1808.04926. our card
  11. Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. 2017. Structured attention networks. arXiv preprint arXiv:1702.00887. — [arXiv:1702.00887](https://arxiv.org/abs/1702.00887) find
  12. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. find
  13. Adam: CoRR, Svetlana Kiritchenko, Berry de Bruijn, Simona Carini, Joel Martin, and Ida Sim. 2010. ExaCT: automatic extraction of clinical trial characteristics from journal publications. BMC medical informatics and decision making, 10(1):56. find
  14. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. arXiv preprint arXiv:1606.04155. our card
  15. M Marre, J Shaw, M Brändle, WM Wan Bebakar, Nor Azmi Kamaruddin, J Strand, M Zdravkovic, TD Le Thi, S Colagiuri, and LEAD-1 SU study group. 2009. Liraglutide, a once-daily human glp-1 analogue, added to a sulphonylurea over 26 weeks produces greater improvements in glycaemic and weight control comp… find
  16. Iain J Marshall, Anna Noel-Storr, Joël Kuiper, James Thomas, and Byron C Wallace. 2018. Machine learning for identifying randomized controlled trials: An evaluation and practitioner’s guide. Research synthesis methods. find
  17. Claudiu Mihăilă, Tomoko Ohta, Sampo Pyysalo, and Sophia Ananiadou. 2013. Biocause: Annotating and analysing causality in the biomedical domain. BMC bioinformatics, 14(1):2. find
  18. Dae Hoon Park and Catherine Blake. 2012. Identifying comparative claim sentences in full-text scientific articles. In Proceedings of the workshop on detecting structure in scholarly discourse, pages 1–9. Association for Computational Linguistics. find
  19. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS-W. find
  20. Sampo Pyysalo, Filip Ginter, Hans Moen, Tapio Salakoski, and Sophia Ananiadou. 2013. Distributional semantics resources for biomedical text processing. Proceedings of Languages in Biology and Medicine. find
  21. Ruty Rinott, Lena Dankin, Carlos Alzate Perez, Mitesh M Khapra, Ehud Aharoni, and Noam Slonim. 2015. Show me your evidence-an automatic method for context dependent evidence detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 440–450. our card
  22. Rodney L Summerscales, Shlomo Argamon, Shangda Bai, Jordan Hupert, and Alan Schwartz. 2011. Automatic summarization of results from clinical trials. In Proceedings of the International Conference on Bioinformatics and Biomedicine (BIBM), pages 372–377. IEEE. find
  23. Paul Thompson, Giulia Venturi, John McNaught, Simonetta Montemagni, and Sophia Ananiadou. 2008. Categorising modality in biomedical texts. In Proceedings of the LREC 2008 Workshop on Building and Evaluating Resources for Biomedical Text Mining, pages 27–34. find
  24. Guy Tsafnat, Adam Dunn, Paul Glasziou, and Enrico Coiera. 2013. The automation of systematic reviews: Would lead to best currently available evidence at the push of a button. BMJ: British Medical Journal, 346(7891):8–8. find
  25. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. our card
  26. Omar Zaidan, Jason Eisner, and Christine D Piatko. 2007. Using” annotator rationales” to improve machine learning for text categorization. In HLT-NAACL, pages 260–267. Citeseer. our card
  27. Ye Zhang, Iain J. Marshall, and Byron C. Wallace. 2016. Rationale-Augmented Convolutional Neural Networks for Text Classification. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 795–804. Associaton for Computational Linguistics (ACL). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lehman-2019-inferring-which-medical-treatments-work/