Elamur Research⌕
☰
home / papers / 2017 / Deep Keyphrase Generation

Deep Keyphrase Generation

Summary

Keyphrases condense what a document is about. Classical extraction pipelines first gather candidate spans, such as n-grams or noun phrases matching POS patterns, and then rank them with statistical features like TF-IDF or PageRank. These methods can only return phrases that literally occur in the text. Yet in four standard datasets (Inspec, Krapivin, NUS, SemEval), roughly a third to well over half of the author-assigned keyphrases never appear verbatim in the abstract. The authors therefore treat the task as generation. An RNN encoder-decoder with a bidirectional GRU encoder reads the source text and produces keyphrases word by word. A copying mechanism, following Gu et al. (2016), lets the decoder reuse rare or out-of-vocabulary words taken directly from the input. The resulting model, CopyRNN, was tested against six baselines on six datasets. It beats both supervised and unsupervised extractors on present keyphrases and also recovers up to about 20% of absent ones, which extractors cannot produce at all. This makes keyphrase assignment closer to semantic summarization, which is useful for indexing and retrieval. Code and data are public.

Key points

  • Share of keyphrases absent from the abstract: Inspec 44.31%, Krapivin 52.26%, NUS 32.25%, SemEval 57.99%.
  • Architecture and decoding:
    • The encoder is a bidirectional GRU and the decoder is an RNN with a copying mechanism.
    • At inference, beam search proposes phrases, and a max heap keeps the most probable word sequences.
  • The authors present this as the first encoder-decoder approach to keyphrase prediction and the first study to target absent keyphrases in scientific publications. They report recall of up to 20% on absent keyphrases.
  • Cross-domain test on DUC-2001 (308 news articles, 2,488 annotated keyphrases), with no retraining:
    • CopyRNN recalled 766 keyphrases, and 14.3% of these contained out-of-vocabulary words, including many names of persons and places.
    • It outperformed TextRank and KeyCluster but trailed the other methods reported by Hasan and Ng (2010).
  • Code and dataset: github.com/memray/seq2seq-keyphrase.

Interesting ideas

  • I1. Flattening one-to-many targets. A document with several keyphrases is turned into several (text, single keyphrase) training pairs, so a standard seq2seq model can be used unchanged. This is an easy trick for any task with multiple gold outputs per input, such as tagging or title suggestion.
  • I2. Separate scores for present and absent outputs. Reporting results separately for gold items that occur verbatim in the input and items that do not shows whether a model actually abstracts or merely extracts. This diagnostic transfers well to summarization or term-extraction evaluations.
  • I3. Copying for rare words. A copy path lets the decoder emit tokens outside its fixed vocabulary, such as names and rare terms. This would be especially valuable for morphologically rich languages like Ukrainian, where vocabulary coverage is a constant problem.
  • I4. Zero-adaptation transfer test. Applying a model trained on scientific abstracts directly to news gives a cheap measure of how much general language knowledge it has learned. The same setup could probe domain robustness before investing in domain-specific annotation.
  • I5. Design inspired by human annotators. The model's two parts mirror how people assign keyphrases: grasping the overall meaning (the encoder) and picking out salient spans from the text (copying). Framing architecture choices around the human annotation process can be a useful heuristic for other annotation-style tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

102 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 100 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1054

@inproceedings{meng2017deep,
  title = {{Deep Keyphrase Generation}},
  author = {Meng, Rui and Zhao, Sanqiang and Han, Shuguang and He, Daqing and Brusilovsky, Peter and Chi, Yu},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1054},
  url = {https://doi.org/10.18653/v1/p17-1054}
}

References

38 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393 . find our card
  2. M. Allamanis, H. Peng, and C. Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. ArXiv e-prints . find
  3. Shuguang Han, Daqing He, Jiepu Jiang, and Zhen Yue. 2013. Supporting exploratory people search: a study of factor transparency and user control. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management. ACM, pages 449–458. find
  4. Zhiyuan Liu, Xinxiong Chen, Yabin Zheng, and Maosong Sun. 2011. Automatic keyphrase extraction by bridging vocabulary gap. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning. Association for Computational Linguistics, pages 135–144. find
  5. Kazi Saidul Hasan and Vincent Ng. 2010. Conundrums in unsupervised keyphrase extraction: making sense of the state-of-the-art. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters. Association for Computational Linguistics, pages 365–373. find
  6. Zhiyuan Liu, Wenyi Huang, Yabin Zheng, and Maosong Sun. 2010. Automatic keyphrase extraction via topic decomposition. In Proceedings of the 2010 conference on empirical methods in natural language processing. Association for Computational Linguistics, pages 366–376. find
  7. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find
  8. Zhiyuan Liu, Peng Li, Yabin Zheng, and Maosong Sun. 2009. Clustering to find exemplar terms for keyphrase extraction. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1. Association for Computational Linguistics, pages 257–266. find
  9. Anette Hulth. 2003. Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 conference on Empirical methods in natural language processing. Association for Computational Linguistics, pages 216–223. find our card
  10. Patrice Lopez and Laurent Romary. 2010. Humb: Automatic key term extraction from scientific articles in grobid. In Proceedings of the 5th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Stroudsburg, PA, USA, SemEval ’10, pages 248–251. http://dl.acm.org/cita… ↗
  11. Anette Hulth and Beáta B Megyesi. 2006. A study on automatically extracted keywords in text categorization. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Association for Computational Li… find
  12. Sumit Chopra Marc’Aurelio Ranzato, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. ICLR, San Juan, Puerto Rico . find our card
  13. Steve Jones and Mark S Staveley. 1999. Phrasier: a system for interactive document retrieval using keyphrases. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval. ACM, pages 160–167. find
  14. Daniel Kelleher and Saturnino Luz. 2005. Automatic hypertext keyphrase detection. In Proceedings of the 19th International Joint Conference on Artificial Intelligence. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, IJCAI’05, pages 1608–1609. http://dl.acm.org/citation.cfm?id=1642293.164257… ↗
  15. Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010. Semeval-2010 task 5: Automatic keyphrase extraction from scientific articles. In Proceedings of the 5th International Workshop on Semantic Evaluation. Association for Computational Linguistics, pages 21–26. find our card
  16. Yutaka Matsuo and Mitsuru Ishizuka. 2004. Keyword extraction from a single document using word co-occurrence statistical information. International Journal on Artificial Intelligence Tools 13(01):157– 169. find
  17. Olena Medelyan, Eibe Frank, and Ian H Witten. 2009a. Human-competitive tagging using automatic keyphrase extraction. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3-Volume 3. Association for Computational Linguistics, pages 1318–1327. find
  18. Olena Medelyan, Eibe Frank, and Ian H. Witten. 2009b. Human-competitive tagging using automatic keyphrase extraction. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3 - Volume 3. Association for Computational Linguistics, Stroudsburg, PA, USA, EMNLP… find
  19. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
  20. Mikalai Krapivin, Aliaksandr Autayeu, and Maurizio Marchese. 2008. Large dataset for keyphrases extraction. Technical Report DISI-09-055, DISI, Trento, Italy. find
  21. Olena Medelyan, Ian H Witten, and David Milne. 2008. Topic indexing with wikipedia. In Proceedings of the AAAI WikiAI workshop. volume 1, pages 19–24. find
  22. Tho Thi Ngoc Le, Minh Le Nguyen, and Akira Shimazu. 2016. Unsupervised Keyphrase Extraction: Introducing New Kinds of Words to Keyphrases, Springer International Publishing, Cham, pages 665–671. find
  23. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into texts. Association for Computational Linguistics. find
  24. Minmei Wang, Bo Zhao, and Yihua Huang. 2016. PTR: Phrase-Based Topical Ranking for Automatic Keyphrase Extraction in Scientific Publications, Springer International Publishing, Cham, pages 120–128. find
  25. Thuy Dung Nguyen and Min-Yen Kan. 2007. Keyphrase extraction in scientific publications. In International Conference on Asian Digital Libraries. Springer, pages 317–326. find
  26. Meng Rui, Han Shuguang, Huang Yun, He Daqing, and Brusilovsky Peter. 2016. Knowledge-based content linking for online textbooks. In 2016 IEEE/WIC/ACM International Conference on Web Intelligence. The Institute of Electrical and Electronics Engineers, pages 18–25. find
  27. Ian H Witten, Gordon W Paynter, Eibe Frank, Carl Gutwin, and Craig G Nevill-Manning. 1999. Kea: Practical automatic keyphrase extraction. In Proceedings of the fourth ACM conference on Digital libraries. ACM, pages 254–255. find
  28. Wenyuan Zeng, Wenjie Luo, Sanja Fidler, and Raquel Urtasun. 2016. Efficient summarization with read-again and copy mechanism. arXiv preprint arXiv:1611.03382 . — [arXiv:1611.03382](https://arxiv.org/abs/1611.03382) find
  29. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015. pages 379–389. http://aclweb.org/a… ↗
  30. Qi Zhang, Yang Wang, Yeyun Gong, and Xuanjing Huang. 2016. Keyphrase extraction using deep recurrent neural networks on twitter. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 836–845. https:/… find
  31. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16). find our card
  32. Yongzheng Zhang, Nur Zincir-Heywood, and Evangelos Milios. 2004. World wide web site summarization. Web Intelligence and Agent Systems: An International Journal 2(1):39–53. find
  33. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, B… find our card
  34. Min Song, Il-Yeol Song, and Xiaohua Hu. 2003. Kpspotter: a flexible information gain-based keyphrase extraction system. In Proceedings of the 5th ACM international workshop on Web information and data management. ACM, pages 50–53. find
  35. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112. find
  36. Takashi Tomokiyo and Matthew Hurst. 2003. A language model approach to keyphrase extraction. In Proceedings of the ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment - Volume 18. Association for Computational Linguistics, Stroudsburg, PA, USA, MWE ’03, pages 33– 40. http… find
  37. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems. pages 2773–2781. find
  38. Xiaojun Wan and Jianguo Xiao. 2008. Single document keyphrase extraction using neighborhood knowledge. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meng-2017-deep-keyphrase-generation/