Elamur Research⌕
☰
home / papers / 2019 / Induction Networks for Few-Shot Text Classification

Induction Networks for Few-Shot Text Classification

Paper · 2019 · Ruiying Geng, Binhua Li, Yongbin Li et al.

Summary

Few-shot text classification requires recognizing new classes from only a handful of labeled examples. Metric-based meta-learners usually compare a query against individual support samples or against a simple sum or average of them. This leaves them vulnerable to the varied phrasings found within a single class. Geng et al. propose Induction Networks, which build a class-level representation from the support set before any comparison is made. The architecture has three parts. An Encoder Module embeds the texts. An Induction Module maps the sample vectors into a class space through a matrix transformation and then aggregates them with capsule-style dynamic routing. A Relation Module then scores how well each query matches each induced class vector. The system is meta-trained end-to-end on randomly sampled episodes. It was tested on the English sentiment benchmark ARSC and on ODIC, a real-world Chinese dialogue intent dataset, where it beat prior state-of-the-art few-shot models. The work shows that inducing robust class representations, rather than matching individual samples, helps models generalize to unseen classes.

Key points

  • Baselines include Matching Networks, Prototypical Networks, Graph Network, Relation Network, SNAIL and ROBUSTTC-FSL. On ODIC the gain over SNAIL is statistically significant at the 99% level under a paired test.
  • The margin over baselines is larger in 10-shot than in 5-shot settings, since averaging-based methods suffer more from sample-level noise as the support set grows.
  • In the ARSC ablation, Routing+Relation reaches 85.63 accuracy with 3 routing iterations, compared with 85.41 for 2 iterations and 85.06 for 1. Further iterations brought no gain.
  • Replacing components lowers ARSC accuracy:
    • Routing+Cosine: 84.67
    • Attention+Relation: 85.15
    • Sum+Relation: 83.07
  • t-SNE plots on ODIC (5-way 10-shot) show two effects. Support vectors become more separable after the matrix transformation. Query vectors are also better clustered than those learned by Relation Network.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China64 works · 14,113 citations🎓Beijing Institute of TechnologyBeijing · China15 works · 3,480 citations🎓Queen's UniversityKingston · Canada8 works · 1,610 citations🎓Queens UniversityDhaka · Bangladesh2 works · 577 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian and Jian Sun. 2019. Induction Networks for Few-Shot Text Classification. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1403

@inproceedings{geng2019induction,
  title = {{Induction Networks for Few-Shot Text Classification}},
  author = {Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian and Jian Sun},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1403},
  url = {https://doi.org/10.18653/v1/d19-1403}
}

References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 440–447.
  2. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159. find
  3. Li Fe-Fei et al. 2003. A bayesian approach to unsupervised one-shot learning of object categories. In Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on, pages 1134–1141. IEEE. find
  4. Li Fei-Fei, Rob Fergus, and Pietro Perona. 2006. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594–611. find
  5. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1126–1135. JMLR. org. find
  6. Victor Garcia and Joan Bruna. 2017. Few-shot learning with graph neural networks. CoRR, abs/1711.04043. — [arXiv:1711.04043](https://arxiv.org/abs/1711.04043) find
  7. Ruiying Geng, Ping Jian, Yingxue Zhang, and Heyan Huang. 2017. Implicit discourse relation identification based on tree structure neural network. In 2017 International Conference on Asian Language Processing (IALP), pages 334–337. IEEE. find
  8. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315– 323. find
  9. Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. 2011. Transforming auto-encoders. In International Conference on Artificial Neural Networks, pages 44–51. Springer. find
  10. Shaohui Kuang, Junhui Li, António Branco, Weihua Luo, and Deyi Xiong. 2018. Attention focusing for neural machine translation by bridging source and target embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages… find
  11. Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. 2018. Analogical reasoning on chinese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 138–143. find
  12. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130.
  13. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. find
  14. Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. 2018. A simple neural attentive meta-learner. In Proceedings of ICLR. find
  15. Tsendsuren Munkhdalai and Hong Yu. 2017. Meta networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2554–2563. JMLR. org. find
  16. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  17. Hang Qi, Matthew Brown, and David G Lowe. 2018. Low-shot learning with imprinted weights. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5822–5830. find
  18. Anthony Rios and Ramakanth Kavuluru. 2018. Few-shot and zero-shot multi-label learning for structured label spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3132–3142.
  19. Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In Advances in Neural Information Processing Systems, pages 3856–3866. find
  20. Justin Salamon and Juan Pablo Bello. 2017. Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Processing Letters, 24(3):279–283. find
  21. Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4077–4087. find
  22. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems, pages 926–934.
  23. Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1199–1208. find
  24. Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pages 3630–3638. find
  25. Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2016. A deep architecture for semantic matching with multiple positional sentence representations. In AAAI, volume 16, pages 2835–2841. find
  26. Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip Yu. 2018. Zero-shot user intent detection via capsule neural networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3090–3099.
  27. Hu Xu, Bing Liu, Lei Shu, and P Yu. 2019. Open-world learning and application to product classification. In The World Wide Web Conference, pages 3413–3419. ACM. find
  28. Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2018. Lifelong domain word embedding via meta-learning. arXiv preprint arXiv:1805.09991. — [arXiv:1805.09991](https://arxiv.org/abs/1805.09991) find
  29. Min Yang, Wei Zhao, Jianbo Ye, Zeyang Lei, Zhou Zhao, and Soufei Zhang. 2018. Investigating capsule networks with dynamic routing for text classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3110–3119. find
  30. Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: H…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/geng-2019-induction-networks-few-shot-text/