Elamur Research
☰
home / papers / 2020 / SPECTER: Document-level Representation Learning using Citation-informed Transformers

SPECTER: Document-level Representation Learning using Citation-informed Transformers

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2004.07180

Summary

Pretrained Transformers such as BERT and SciBERT learn from token- and sentence-level objectives. As a result, passing a paper's title and abstract through them unchanged gives weak whole-document vectors for tasks like topic classification or recommendation. The authors propose SPECTER, which starts from SciBERT and continues training with a triplet margin loss derived from the citation graph. The loss pushes a query paper closer, in L2 distance, to a paper it cites than to one it does not. Negatives mix random papers with "hard" ones: works cited by a cited paper but not by the query itself. The embedding is the final [CLS] vector over the concatenated title and abstract. Citations are therefore needed only during training, and uncited new papers can still be embedded. The authors also release SciDocs, a seven-task benchmark covering citation prediction, user-activity prediction, classification and recommendation. Used as fixed features without fine-tuning, SPECTER beats a range of strong baselines, including graph-based SGC and task-fine-tuned SciBERT. The result is a reusable paper encoder plus an open evaluation suite.

Key points

  • Training setup: The input is [CLS] + title + [SEP] + abstract (WordPieces), with SciBERT as initialization. The triplet loss uses margin m = 1 and L2 distance; cosine-style distances performed worse.
  • Ablations: Adding venue or author fields gave no benefit, and encoding title and abstract separately was worse. Including hard negatives produced better embeddings than using random negatives alone.
  • SciDocs benchmark: It covers 7 tasks. One example is MeSH classification over 23K medical papers in 11 top-level disease classes. Another is a set of anonymized click logs gathered from a web-based recommendation system. It was motivated by older benchmarks being small and already saturated (99%+ AUC).
  • Comparison with fine-tuning: SPECTER without fine-tuning outperformed SciBERT fine-tuned on the end tasks, both per task and in a multitask setup. Further task-specific fine-tuning of SPECTER brought no extra gains.
  • Released resources: The model and code (github.com/allenai/specter), the SciDocs data (github.com/allenai/scidocs) and a training set of hundreds of thousands of titles, abstracts and citations are public. Exact benchmark scores were in the part of the paper omitted from the provided text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

215 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 208 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.207

@inproceedings{cohan2020specter,
  title = {{SPECTER: Document-level Representation Learning using Citation-informed Transformers}},
  author = {Cohan, Arman and Feldman, Sergey and Beltagy, Iz and Downey, Doug and Weld, Daniel},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.207},
  url = {https://doi.org/10.18653/v1/2020.acl-main.207}
}

References

58 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anant K. Agarwal, Ivan Zaitsev, Xuanhui Wang, Cheng Yen Li, Marc Najork, and Thorsten Joachims. 2019. Estimating position bias without intrusive interventions. In WSDM. find
  2. Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler C. Murray, Hsu-Han Ooi, Matthew E. Peters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Christophe… find
  3. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. find
  4. A simple but tough-to-beat baseline for sentence embeddings. In ICLR. find
  5. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A Pretrained Language Model for Scientific Text. In EMNLP. find
  6. Chandra Bhagavatula, Sergey Feldman, Russell Power, and Waleed Ammar. 2018. Content-Based Citation Recommendation. In NAACL-HLT. find
  7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL. find
  8. Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral networks and locally connected networks on graphs. ICLR. find
  9. Liqun Chen, Guoyin Wang, Chenyang Tao, Dinghan Shen, Pengyu Cheng, Xinyuan Zhang, Wenlin Wang, Yizhe Zhang, and Lawrence Carin. 2019. Improving textual network embedding with global attention via optimal transport. In ACL. find
  10. Tsung Teng Chen and Maria Lee. 2018. Research Paper Recommender Systems on Big Scholarly Data. In Knowledge Management and Acquisition for Intelligent Systems. find
  11. K. Bretonnel Cohen, Helen L. Johnson, Karin M. Verspoor, Christophe Roeder, and Lawrence Hunter. 2010. The structural and content aspects of abstracts versus bodies of full text journal articles are different. BMC Bioinformatics, 11:492–492. find
  12. Christophe Van Gysel, Maarten de Rijke, and Evangelos Kanoulas. 2017. Neural Vector Spaces for Unsupervised Information Retrieval. ACM Trans. Inf. Syst. find
  13. Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017a. Inductive Representation Learning on Large Graphs. In NIPS. find our card
  14. William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017b. Inductive representation learning on large graphs. In NIPS. find our card
  15. Erik Holmer and Andreas Marfurt. 2018. Explaining away syntactic structure in semantic document representations. ArXiv, abs/1806.01620. — [arXiv:1806.01620](https://arxiv.org/abs/1806.01620) find
  16. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In ACL. find
  17. Chanwoo Jeong, Sion Jang, Hyuna Shin, Eunjeong Lucy Park, and Sungchul Choi. 2019. A context-aware citation recommendation model with bert and graph convolutional networks. ArXiv, abs/1903.06464. — [arXiv:1903.06464](https://arxiv.org/abs/1903.06464) find
  18. Anshul Kanakia, Zhihong Shen, Darrin Eide, and Kuansan Wang. 2019. A Scalable Hybrid Research Paper Recommender System for Microsoft Academic. In WWW. find
  19. Diederik P. Kingma and Jimmy Ba. 2014. A Method for Stochastic Optimization. abs/1412.6980. find
  20. Adam: ArXiv, Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In EMNLP. find our card
  21. Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. ICLR. find our card
  22. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. find
  23. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In NIPS. find
  24. Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. 1996. A Density-based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In KDD. find
  25. Richard Klavans and Kevin W. Boyack. 2006. Identifying a better measure of relatedness for mapping science. Journal of the Association for Information Science and Technology, 57:251–263. find
  26. Sergey Feldman, Waleed Ammar, Kyle Lo, Elly Trep-man, Madeleine van Zuylen, and Oren Etzioni. 2019. Quantifying Sex Bias in Clinical Studies at Scale With Automated Data Extraction. JAMA. find
  27. Jey Han Lau and Timothy Baldwin. 2016. An empirical evaluation of doc2vec with practical insights into document embedding generation. In Rep4NLP@ACL. find
  28. J Ganesh, Manish Gupta, and Vijay K. Varma. 2016. Doc2sent2vec: A novel two-phase approach for learning document representation. In SIGIR. find
  29. Quoc Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. In ICML. find our card
  30. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A Deep Semantic Natural Language Processing Platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS). find our card
  31. Jimmy J. Lin. 2008. Is searching full text more effective than searching abstracts? BMC Bioinformatics, 10:46–46. find
  32. Carolyn E Lipscomb. 2000. Medical Subject Headings (MeSH). Bulletin of the Medical Library Association. find
  33. Chundi Liu, Shunan Zhao, and Maksims Volkovs. 2018. Unsupervised Document Embedding with CNNs. ArXiv, abs/1711.04168v3. — [arXiv:1711.04168](https://arxiv.org/abs/1711.04168) find
  34. Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang. 2015. An Overview of Microsoft Academic Service (MAS) and Applications. In WWW. find
  35. Pengfei Liu, King Keung Wu, and Helen M. Meng. 2017. A Model of Extended Paragraph Vector for Document Categorization and Trend Analysis. IJCNN. find
  36. Cunchao Tu, Han Liu, Zhiyuan Liu, and Maosong Sun. 2017. Cane: Context-aware network embedding for relation modeling. In ACL. find
  37. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar S. Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv, abs/1907.11692. find
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In NIPS. find
  39. Laurens van der Maaten. 2014. Accelerating t-SNE Using Tree-based Algorithms. Journal of Machine Learning Research. find
  40. Allen Nie, Erin Bennett, and Noah Goodman. 2019. DisSent: Learning Sentence Representations from Explicit Discourse Relations. In ACL. find
  41. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  42. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. find
  43. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. arXiv. find
  44. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In LREC. find
  45. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In EMNLP. find
  46. Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A Conditional Entropy-based External Cluster Evaluation Measure. In EMNLP. find our card
  47. J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. 2007. Collaborative filtering recommender systems. In The adaptive web. Springer. find
  48. Martijn J. Schuemie, Marc Weeber, Bob J. A. Schijve-naars, Erik M. van Mulligen, C. Christiaan van der Eijk, Rob Jelier, Barend Mons, and Jan A. Kors. 2004. Distribution of information in biomedical abstracts and full-text publications. Bioinformatics, 20(16):2597–604. find
  49. Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018. Improved semantic-aware network embedding with fine-grained word alignment. In EMNLP. find
  50. Wenlin Wang, Chenyang Tao, Zhe Gan, Guoyin Wang, Liqun Chen, Xinyuan Zhang, Ruiyi Zhang, Qian Yang, Ricardo Henao, and Lawrence Carin. 2019. Improving textual network learning with variational homophilic embeddings. In Advances in Neural Information Processing Systems, pages 2074–2085. find
  51. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In NAACL-HLT. find
  52. Felix Wu, Amauri H. Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. 2019a. Simplifying graph convolutional networks. In ICML. find
  53. Lingfei Wu, Ian En-Hsu Yen, Kun Xu, Fangli Xu, Avinash Balakrishnan, Pin-Yu Chen, Pradeep Ravikumar, and Michael J Witbrock. 2018. Word Mover’s Embedding: From Word2Vec to Document Embedding. In EMNLP. find
  54. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. ArXiv, abs/1609.08144. find
  55. Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. 2019b. A Comprehensive Survey on Graph Neural Networks. ArXiv, abs/1901.00596. find our card
  56. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. ArXiv, abs/1906.08237. find our card
  57. Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik G. Learned-Miller, and Jaap Kamps. 2018. From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing. In CIKM. find
  58. Xinyuan Zhang, Yitong Li, Dinghan Shen, and Lawrence Carin. 2018. Diffusion maps for textual network embedding. In NeurIPS. find