Elamur Research⌕
☰
home / papers / 2018 / SGM: Sequence Generation Model for Multi-label Classification

SGM: Sequence Generation Model for Multi-label Classification

Summary

Multi-label classification (MLC) assigns several labels to one text. Common approaches such as binary relevance predict each label independently, and earlier neural models rarely account for the fact that different words matter for different labels. Yang et al. recast MLC as generating a sequence of labels. Their SGM model reads the text with a bidirectional LSTM. An attention-equipped LSTM decoder then emits labels one at a time, so each prediction depends on the labels already produced and on the most relevant parts of the input. Labels are ordered by training-set frequency, a masked softmax prevents repeats, and beam search selects the final output. Early mistakes can cascade through later predictions (exposure bias). To limit this, the authors add a "global embedding," a gated blend of the top label's embedding and a probability-weighted average of all label embeddings. On RCV1-V2 and AAPD, SGM clearly outperforms prior methods. Analysis shows it learns label correlations and selects informative words. The work shows how Seq2Seq modeling can handle classification tasks with interdependent labels.

Key points

  • Datasets:
    • RCV1-V2 has 804,414 Reuters newswire stories, 103 labels, about 3.24 labels and 124 words per sample.
    • AAPD has 55,840 samples, 54 labels, about 2.41 labels and 163 words per sample.
    • The provided extract omits the score tables, so no metric values are given here.
  • Architecture:
    • BiLSTM encoder with additive attention.
    • LSTM decoder that uses the previous label embedding and context vector.
    • Masked softmax (−∞ for labels already predicted).
    • Cross-entropy training and beam search at inference.
  • Global embedding: a transform gate inspired by highway networks combines the greedy label embedding with the expected embedding over the full predicted distribution. This reduces damage from earlier mispredictions.
  • Qualitative analysis:
    • SGM recovers highly correlated label pairs that BR misses, such as E51 and E512 (correlation 0.7664).
    • Samples with many gold labels remain difficult.
  • Availability: code and data are at github.com/lancopku/SGM. Published at COLING 2018, pp. 3915–3926.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

93 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 91 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yang2018sgm,
  title = {{SGM: Sequence Generation Model for Multi-label Classification}},
  author = {Yang, Pengcheng and Sun, Xu and Li, Wei and Ma, Shuming and Wu, Wei and Wang, Houfeng},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  url = {https://aclanthology.org/C18-1330/}
}

References

40 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
  2. Simon Baker and Anna Korhonen. 2017. Initializing neural networks for hierarchical multi-label text classification. In BioNLP. find
  3. Fernando Benites and Elena Sapozhnikova. 2015. Haram: a hierarchical aram neural network for large-scale text classification. In Data Mining Workshop (ICDMW), 2015 IEEE International Conference on, pages 847–854. IEEE. find
  4. Matthew R. Boutell, Jiebo Luo, Xipeng Shen, and Christopher M. Brown. 2004. Learning multi-label scene classification. Pattern Recognition, 37(9):1757–1771. find
  5. Guibin Chen, Deheng Ye, Zhenchang Xing, Jieshan Chen, and Erik Cambria. 2017. Ensemble application of convolutional and recurrent neural networks for multi-label text categorization. In 2017 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, AK, USA, May 14-19, 2017, pages 237… find
  6. Amanda Clare and Ross D King. 2001. Knowledge discovery in multi-label phenotype data. In European Conference on Principles of Data Mining and Knowledge Discovery, pages 42–53. Springer. find
  7. André Elisseeff and Jason Weston. 2002. A kernel method for multi-labelled classification. In Advances in neural information processing systems, pages 681–687. find
  8. Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Mencı́a, and Klaus Brinker. 2008. Multilabel classification via calibrated label ranking. Machine learning, 73(2):133–153. find
  9. Nadia Ghamrawi and Andrew McCallum. 2005. Collective multi-label classification. In Proceedings of the 14th ACM international conference on Information and knowledge management, pages 195–200. ACM. find
  10. Siddharth Gopal and Yiming Yang. 2010. Multilabel classification with meta-level features. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 315–322. ACM. find
  11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  12. Ioannis Katakis, Grigorios Tsoumakas, and Ioannis Vlahavas. 2008. Multilabel text classification for automated tag suggestion. In Proceedings of the ECML/PKDD, volume 18. find
  13. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746–1751. find our card
  14. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980. find
  15. Gakuto Kurata, Bing Xiang, and Bowen Zhou. 2016. Improved neural network-based multi-label classification with better initialization leveraging label co-occurrence. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T… find
  16. David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361–397. find our card
  17. Li Li, Houfeng Wang, Xu Sun, Baobao Chang, Shi Zhao, and Lei Sha. 2015. Multi-label text categorization with joint learning predictions-as-features method. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 835–839. find
  18. Junyang Lin, Xu Sun, Shuming Ma, and Qi Su. 2018. Global encoding for abstractive summarization. In ACL 2018. find
  19. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. CoRR, abs/1508.04025. find
  20. Shuming Ma, Xu Sun, Yizhong Wang, and Junyang Lin. 2018. Bag-of-words as target for neural machine translation. In ACL 2018. find
  21. Christopher D Manning, Prabhakar Raghavan, Hinrich Schütze, et al. 2008. Introduction to information retrieval, volume 1. Cambridge university press Cambridge. find
  22. Jinseok Nam, Jungi Kim, Iryna Gurevych, and Johannes Fürnkranz. 2013. Large-scale multi-label text classification - revisiting neural networks. CoRR, abs/1312.5419. — [arXiv:1312.5419](https://arxiv.org/abs/1312.5419) find
  23. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pages 1310–1318. find
  24. Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. 2011. Classifier chains for multi-label classification. Machine learning, 85(3):333. find
  25. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 379–389. find
  26. Robert E Schapire and Yoram Singer. 1999. Improved boosting algorithms using confidence-rated predictions. Machine learning, 37(3):297–336. find
  27. Robert E Schapire and Yoram Singer. 2000. Boostexter: A boosting-based system for text categorization. Machine learning, 39(2-3):135–168. find
  28. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. CoRR, abs/1705.09655. find our card
  29. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929– 1958. find
  30. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. abs/1505.00387. 2015. find our card
  31. Highway networks. CoRR, Xu Sun, Bingzhen Wei, Xuancheng Ren, and Shuming Ma. 2017. Label embedding network: Learning label representation for soft training of deep networks. CoRR, abs/1710.10393. — [arXiv:1710.10393](https://arxiv.org/abs/1710.10393) find
  32. Piotr Szymański, Tomasz Kajdanowicz, and Kristian Kersting. 2016. How is a data-driven approach better than random choice in label space division for multi-label classification? Entropy, 18(8):282. find
  33. Piotr Szymański. 2017. A scikit-based python environment for performing multi-label classification. arXiv preprint arXiv:1702.01460. — [arXiv:1702.01460](https://arxiv.org/abs/1702.01460) find
  34. Grigorios Tsoumakas and Ioannis Katakis. 2006. Multi-label classification: An overview. International Journal of Data Warehousing and Mining, 3(3). find
  35. Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2011. Random k-labelsets for multilabel classification. IEEE Transactions on Knowledge and Data Engineering, 23(7):1079–1089. find
  36. Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. CoRR, abs/1606.02960. find
  37. Jingjing Xu, Xu Sun, Qi Zeng, Xuancheng Ren, Xiaodong Zhang, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. find our card
  38. Ye Zhang and Byron C. Wallace. 2015. A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. CoRR, abs/1510.03820. find our card
  39. Min-Ling Zhang and Zhi-Hua Zhou. 2006. Multilabel neural networks with applications to functional genomics and text categorization. IEEE Transactions on Knowledge and Data Engineering, 18(10):1338–1351. find
  40. Min-Ling Zhang and Zhi-Hua Zhou. 2007. ML-KNN: A lazy learning approach to multi-label learning. Pattern recognition, 40(7):2038–2048. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2018-sgm-sequence-generation-model-multi/