Elamur Research⌕
☰
home / papers / 2018 / Measuring the Evolution of a Scientific Field through Citation Frames

Measuring the Evolution of a Scientific Field through Citation Frames

Summary

Citation counts usually treat every reference as equal, but authors cite earlier work for very different purposes. Before this paper, those purposes had only been studied by hand in small samples. Jurgens and colleagues study citation function across an entire discipline, NLP. They merge earlier taxonomies into six classes: BACKGROUND, MOTIVATION, USES, EXTENSION, COMPARISON OR CONTRAST and FUTURE. With this scheme they annotate 1,969 citation contexts from the ACL Anthology Reference Corpus (ARC). The data trains a state-of-the-art classifier that combines several feature types: where a citation sits in the paper, lexical and grammatical patterns, field-level metadata, and new unsupervised features drawn from topic models and argument structure. The classifier then labels over 134,000 citations in more than 20,000 papers covering almost forty years. The results show that citation types track a paper's discourse structure but differ by venue, and that a paper's framing helps forecast its later citation count. Over time, NLP authors compare less and reuse a shared core of methods, which signals rising consensus. The released data and code support large-scale studies of how science develops.

Key points

  • Dataset: 1,969 annotated citations in total.
    • 1,436 come from 52 fully labeled ARC papers.
    • 533 come from 133 further papers, gathered by keyword-targeted sampling of rare classes.
    • BACKGROUND is by far the largest class (1,021), and FUTURE is the smallest (68).
  • Annotation pipeline:
    • Citations and their references were extracted with ParsCit.
    • Each instance was labeled by two NLP-trained annotators in Brat.
    • All disagreements were fully adjudicated.
  • Field-scale labeling: 134,127 citations from over 20,000 NLP papers were classified automatically.
  • Consensus trend: COMPARISON OR CONTRAST citations decline steadily (r = −0.899), and non-methodological citations decline overall (r = −0.663). Comparisons and uses also concentrate on a smaller set of works. One exception is a small rise after ACL venues allowed unlimited references (ACL in 2010, NAACL in 2011).
  • Resources: The data and code are released at github.com/davidjurgens/citation-function.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

101 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 95 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00028

@article{jurgens2018measuring,
  title = {{Measuring the Evolution of a Scientific Field through Citation Frames}},
  author = {Jurgens, David and Kumar, Srijan and Hoover, Raine and McFarland, Dan and Jurafsky, Dan},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00028},
  url = {https://doi.org/10.1162/tacl_a_00028}
}

References

73 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amjad Abu-Jbara and Dragomir Radev. 2012. Reference scope identification in citing sentences. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 80–90. Association for Computational Ling… find
  2. Amjad Abu-Jbara, Jefferson Ezra, and Dragomir R. Radev. 2013. Purpose and polarity of citation: Towards NLP-based bibliometrics. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 596–60… find
  3. Hirotugu Akaike. 1974. A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6):716–723. find
  4. Ashton Anderson, Dan McFarland, and Dan Jurafsky. 2012. Towards a computational history of the ACL: 1980-2008. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 13–21. find
  5. Awais Athar and Simone Teufel. 2012. Context-enhanced citation sentiment detection. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 597–601. Association for Computational Linguistics. find
  6. Awais Athar. 2014. Sentiment analysis of scientific citations. Technical Report, University of Cambridge, Computer Laboratory. Marc Bertin, Iana Atanassova, Yves Gingras, and Vincent Larivière. 2016. The invariant distribution of references in scientific articles. Journal of the Association for Inf… find
  7. Steven Bird, Robert Dale, Bonnie J. Dorr, Bryan R. Gibson, Mark Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir R. Radev, and Yee Fan Tan. 2008. The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics. In Proceedings of International C… find our card
  8. Terrence A. Brooks. 1986. Evidence of complex citer motivations. Journal of the American Society for Information Science, 37(1):34–36. find
  9. Sabine Buchholz, Jorn Veenstra, and Walter Daelemans. 1999. Cascaded grammatical relation assignment. In Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 239–246. find
  10. Donald O. Case and Georgeann M. Higgins. 2000. How can we investigate citation behavior? A study of reasons for citing literature in communication. Journal of the American Society for Information Science, 51(7):635–645. find
  11. Tanmoy Chakraborty, Suhansanu Kumar, Pawan Goyal, Niloy Ganguly, and Animesh Mukherjee. 2014. Towards a stratified learning approach to predict future citation counts. In Proceedings of the 14th Joint Conference on Digital Libraries (JCDL), pages 351–360. find
  12. Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321–357. find our card
  13. Dale E. Chubin and Soumyo D. Moitra. 1975. Content analysis of references: Adjunct or alternative to citation counting? Social Studies of Science, 5:423–441. find
  14. Randall Collins. 1994. Why the social sciences won’t become high-consensus, rapid-discovery science. Sociological Forum, 9(2):155–177. find
  15. Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of the Eighth Conference on European Chapter of the Association for Computational Linguistics (EACL), pages 16–23. find our card
  16. Michael Collins. 1999. Head-driven statistical models for natural language parsing. Ph.D. thesis, University of Pennsylvania. find
  17. Isaac G. Councill, C. Lee Giles, and Min-Yen Kan. 2008. ParsCit: An open-source CRF reference string parsing package. In Proceedings of International Conference on Language Resources and Evaluation (LREC). Hal Daumé III. 2016. Workshops and mini-conferences, November. https://nlpers.blogspot.com/2… ↗
  18. Ying Ding, Xiaozhong Liu, Chun Guo, and Blaise Cronin. 2013. The distribution of references across texts: Some implications for citation analysis. Journal of Informetrics, 7(3):583–592. find
  19. Ying Ding, Guo Zhang, Tamy Chambers, Min Song, Xiaolong Wang, and Chengxiang Zhai. 2014. Content-based citation analysis: The next generation of citation analysis. Journal of the Association for Information Science and Technology (JASIST), 65(9):1820–1833. Cailing Dong and Ulrich Schäfer. 2011. Ens… find
  20. Yuxiao Dong, Reid A. Johnson, and Nitesh V. Chawla. 2016. Can scientific impact be predicted? IEEE Transactions on Big Data (TBD), 2(1):18–30. find
  21. Katrin Erk. 2007. A simple, similarity-based model for selectional preferences. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), pages 216–223. find
  22. Eliza D. Evans, Charles J. Gomez, and Daniel A. Mc-Farland. 2016. Measuring paradigmaticness of disciplines using text. Sociological Science, 3:757–778. Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim. 2014. Do we need hundreds of classifiers to solve real world classificati… find
  23. James H. Fowler and Dag W. Aksnes. 2007. Does self-citation pay? Scientometrics, 72(3):427–437. find
  24. Eugene Garfield. 1979. Citation Indexing Its Theory and Application in Science, Technology, and Humanities. find
  25. John Wiley & Sons, New York. Mark Garzone and Robert E. Mercer. 2000. Towards an automated citation classifier. In Proceedings of the 13th Biennial Conference of the Canadian Society on Computational Studies of Intelligence, pages 337–346. find
  26. Ali Gazni and Fereshteh Didegah. 2011. Investigating different types of research collaboration and citation impact: a case study of Harvard University’s publications. Scientometrics, 87(2):251–265. find
  27. Erving Goffman. 1974. Frame analysis: An essay on the organization of experience. find
  28. Harvard University Press. John J. Gumperz. 1982. Discourse strategies, volume 1. find
  29. Cambridge University Press. Andrew Haas. 1989. A parsing algorithm for unification grammar. Computational Linguistics, 15(4):219–232. find
  30. David Hall, Daniel Jurafsky, and Christopher D. Manning. 2008. Studying the history of ideas using topic models. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 363–371. find
  31. Nigel Harwood. 2009. An interview-based study of the functions of citations in academic writing across two disciplines. Journal of Pragmatics, 41(3):497–518. find
  32. Qi He, Daniel Kifer, Jian Pei, Prasenjit Mitra, and C. Lee Giles. 2011. Citation recommendation without author supervision. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (WSDM), pages 755–764. ACM. Myriam Hernández-Alvarez and Josém Gomez. 2016. Survey abo… find
  33. Zhigang Hu, Chaomei Chen, and Zeyuan Liu. 2013. Where are citations located in the body of scientific articles? A study of the distributions of citation locations. Journal of Informetrics, 7(4):887–896. find
  34. Saurabh Kataria, Prasenjit Mitra, Cornelia Caragea, and C. Lee Giles. 2011. Context sensitive topic models for author influence in document networks. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), volume 22, page 2274. find
  35. Citeseer. Srijan Kumar. 2016. Structure and dynamics of signed citation networks. In Proceedings of the 25th International Conference Companion on World Wide Web (WWW). find
  36. Michael H. Kutner, Chris Nachtsheim, and John Neter. 2004. Applied Linear Regression Models. McGraw-Hill/Irwin. find
  37. Bruno Latour. 1987. Science in action: How to follow scientists and engineers through society. find
  38. Harvard University Press. Geoffrey Leech. 1992. 100 million words of English: The British National Corpus (BNC). Language Research, 28(1):1–13. find
  39. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of the System Demonstrations at 52th Annual Meeting of the Association for Computational Linguistics (ACL), pages… find our card
  40. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  41. Robert E. Mercer, Chrysanne Di Marco, and Frederick W. Kroon. 2004. The frequency of hedging cues in citation contexts in scientific writing. In Conference of the Canadian Society for Computational Studies of Intelligence, pages 75–88. find
  42. Springer. James Moody. 2004. The structure of a social science collaboration network: Disciplinary cohesion from 1963 to 1999. American Sociological Review, 69(2):213–238. find
  43. Michael J. Moravcsik and Poovanalingam Murugesan. 1975. Some results on the function and quality of citations. Social Studies of Science, 5(1):86–92. find
  44. Hidetsugu Nanba and Manabu Okumura. 1999. Towards multi-paper summarization using reference information. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 926–931. find
  45. Kevin Ngozi Nwogu. 1997. The medical research paper: Structure and functions. English for Specific Purposes, 16(2):119–138. find
  46. Robert M. O’brien. 2007. A caution regarding rules of thumb for variance inflation factors. Quality & Quantity, 41(5):673–690. find
  47. Charles Oppenheim and Susan P. Renn. 1978. Highly cited old papers and the reasons why they continue to be cited. Journal of the American Society for Information Science, 29(5):227–231. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu B… find
  48. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  49. Fernando C.N. Pereira and David H.D. Warren. 1983. Parsing as deduction. In Proceedings of the 21st annual meeting on Association for Computational Linguistics (ACL), pages 137–144. find
  50. Bluma C. Peritz. 1983. A classification of citation roles for the social sciences and related fields. Scientometrics, 5(5):303–312. find
  51. S. B. Pham and A. Hoffmann. 2003. A new approach for scientific citation classification using cue phrases. In Proceedings of the Australian Joint Conference in Artificial Intelligence (AI), pages 759–771. find
  52. Ellen Riloff and Rosie Jones. 1999. Learning dictionaries for information extraction by multi-level bootstrapping. In Proceedings of Association for the Advancement of Artificial Intelligence (AAAI), pages 474–479. find
  53. Anna Ritchie, Stephen Robertson, and Simone Teufel. 2008. Comparing citation contexts for information retrieval. In Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM), pages 213–222. find
  54. Xiaolin Shi, Jure Leskovec, and Daniel A. McFarland. 2010. Citing for high impact. In Proceedings of the 10th annual Joint Conference on Digital libraries (JCDL), pages 49–58. ACM/IEEE-CS. find
  55. Stuart M Shieber. 1988. A uniform architecture for parsing and generation. In Proceedings of the 12th Conference on Computational Linguistics, pages 614–619. Association for Computational Linguistics. find
  56. John Skelton. 1994. Analysis of the structure of original research papers: An aid to writing original papers for publication. British Journal of General Practice, 44(387):455–459. find
  57. Henry Small. 2011. Interpreting maps of science using citation context sentiments: A preliminary investigation. Scientometrics, 87(2):373–388. Karen Spärck Jones. 2001. Natural language processing: a historical review. University of Cambridge, pages 2–10. Ina Spiegel-Rüsing. 1977. Bibliometric and… find our card
  58. David I. Stern. 2014. High-ranked social science journal articles can be identified from early citation information. PloS One, 9(11):e112520. find
  59. John Swales. 1986. Citation analysis and discourse analysis. Applied linguistics, 7(1):39–56. find
  60. John Swales. 1990. Genre Analysis: English in Academic and Research Settings. Chapter 7: Research articles in English. find
  61. Cambridge University Press, Cambridge, UK. Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006a. An annotation scheme for citation function. In Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, pages 80–87. Association for Computational Linguistics. find
  62. Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006b. Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 103–110. find our card
  63. Simone Teufel. 2000. Argumentative zoning: Information extraction from scientific text. Ph.D. thesis, University of Ediburgh. find
  64. Simone Teufel. 2010. The structure of scientific articles: Applications to Indexing and Summarization. find
  65. CSLI Publications. Marco Valenzuela, Vu Ha, and Oren Etzioni. 2015. Identifying meaningful citations. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 21–26. find
  66. Peter Vinkler. 1998. Comparative investigation of frequency and strength of motives toward referencing. the reference threshold model. Scientometrics, 43(1):107–127. find
  67. Xiaojun Wan and Fang Liu. 2014. Are all literature citations equally important? Automatic citation strength estimation and its applications. Journal of the Association for Information Science and Technology (JA-SIST), 65(9):1929–1938. Dashun Wang, Chaoming Song, and Albert-László Barabási. 2013.… find
  68. Howard D. White. 2004. Citation analysis and discourse analysis revisited. Applied linguistics, 25(1):89–116. find
  69. Rui Yan, Jie Tang, Xiaobing Liu, Dongdong Shan, and Xiaoming Li. 2011. Citation count prediction: Learning to estimate future citations for literature. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM), pages 1247–1252. find
  70. Rui Yan, Congrui Huang, Jie Tang, Yan Zhang, and Xiaoming Li. 2012. To better stand on the shoulder of giants. In Proceedings of the 12th Joint Conference on Digital Libraries (JCDL), pages 51–60. find
  71. Dani Yogatama, Michael Heilman, Brendan O’Connor, Chris Dyer, Bryan R. Routledge, and Noah A. Smith. 2011. Predicting a scientific community’s response to an article. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 594–604. find
  72. Xiaodan Zhu, Peter Turney, Daniel Lemire, and André Vellino. 2015. Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology, 66(2):408–427. find
  73. John M. Ziman. 1968. Public Knowledge: An Essay Concerning the Social Dimensions of Science. Cambridge University Press, Cambridge, UK. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jurgens-2018-measuring-evolution-scientific-field-through/