Elamur Research⌕
☰
home / papers / 2018 / Diverse Few-Shot Text Classification with Multiple Metrics

Diverse Few-Shot Text Classification with Multiple Metrics

Paper · 2018 · Mo Yu, Xiaoxiao Guo, Jinfeng Yi et al.

Summary

Most few-shot learning research assumes that all tasks are drawn from one large, homogeneous collection, as with Omniglot or ImageNet. Under that assumption, a single shared similarity metric works well. Real text classification workloads are different: tasks span many domains and can have different numbers of labels. For example, an enterprise cloud service may receive many small client datasets. The authors argue that one metric cannot cover this variety.

They propose ROBUSTTC-FSL, which works in two stages:

  • Meta-training. Training tasks are grouped by how well a model trained on one task transfers to another. Missing or unreliable transfer scores are filled in with matrix completion, and the tasks are then partitioned with spectral clustering. Each cluster gets its own CNN-based Matching Network encoder, which serves as that cluster's metric.
  • Meta-testing. For each new few-shot task, the method learns a weighted mix of the cluster metrics.

Experiments on Amazon review sentiment tasks and dialog intent tasks show clear gains over single-metric baselines such as Matching Networks and ProtoNet. This supports keeping several metrics when few-shot tasks are diverse.

Key points

  • The clustering step builds a cross-task transfer-performance matrix S, in which entry (i, j) is the accuracy of adapting task i's model to task j. S is cleaned and completed, then clustered with spectral clustering.
  • Unlike earlier task-clustering methods such as ASAP-MT-LR, ROBUSTTC works with deep networks and with tasks whose label counts differ, as in intent classification.
  • The sentiment tasks are Amazon review domains, each binarized at a star threshold of 2, 4, or 5. A visualization groups the training tasks into 10 clusters, and tasks with the same threshold or similar domains tend to land together.
  • Every tested number of clusters beat the single-metric setting (#cluster = 1). The paper highlights 5 clusters for sentiment and 20 for intent.
  • ROBUSTTC clusters slightly outperform ASAP-MT-LR clusters on sentiment (83.12 vs. 82.65 accuracy with 5 clusters). Code and data are released at github.com/Gorov/DiverseFewShot_Amazon.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States85 works · 66,118 citations🔬IBM Research - Thomas J. Watson Research CenterYorktown Heights · United States41 works · 46,197 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Evaluation & benchmarks · 685 works

Cite

Cite as Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang and Bowen Zhou. 2018. Diverse Few-Shot Text Classification with Multiple Metrics. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1109

@inproceedings{yu2018diverse,
  title = {{Diverse Few-Shot Text Classification with Multiple Metrics}},
  author = {Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang and Bowen Zhou},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1109},
  url = {https://doi.org/10.18653/v1/n18-1109}
}

References

28 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aviad Barzilai and Koby Crammer. 2015. Convex multi-task learning by clustering. In AISTATS. find
  2. Brenden M Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B Tenenbaum. 2011. One shot learning of simple visual concepts. In CogSci, volume 172, page 2. find
  3. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In ACL, volume 7, pages 440–447. our card
  4. Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338. find
  5. Emmanuel J Candès and Terence Tao. 2010. The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory, 56(5):2053–2080. find
  6. Fei-Fei Li, Rob Fergus, and Pietro Perona. 2006. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):594–611. find
  7. Venkat Chandrasekaran, Sujay Sanghavi, Pablo A Parrilo, and Alan S Willsky. 2011. Rank-sparsity incoherence for matrix decomposition. SIAM Journal on Optimization, 21(2):572–596. find
  8. Erik G Miller, Nicholas E Matsakis, and Paul A Viola. 2000. Learning from one example through shared densities on transforms. In Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on, volume 1, pages 464–471. IEEE. find
  9. Yu Cheng, Mo Yu, Xiaoxiao Guo, and Bowen Zhou. 2017. Few-shot learning with meta metric learners. In NIPS 2017 Workshop on Meta-Learning. find
  10. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493–2537. our card
  11. Koby Crammer and Yishay Mansour. 2012. Learning multiple tasks using shared hypotheses. In Advances in Neural Information Processing Systems, pages 1475–1483. find
  12. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400. find
  13. Rie Johnson and Tong Zhang. 2016. Supervised and semi-supervised text categorization using one-hot lstm for region embeddings. stat, 1050:7. find
  14. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. find
  15. Zhuoliang Kang, Kristen Grauman, and Fei Sha. 2011. Learning with whom to share in multi-task feature learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 521–528. find
  16. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746– 1751, Doha, Qatar. Association for Computational Linguistics. our card
  17. Gregory Koch. 2015. Siamese neural networks for one-shot image recognition. Ph.D. thesis, University of Toronto. find
  18. Abhishek Kumar and Hal Daume III. 2012. Learning task grouping and overlap in multi-task learning. In Proceedings of the 29th International Conference on Machine Learning (ICML-12). find
  19. Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. 2017. A simple neural attentive meta-learner. In NIPS 2017 Workshop on Meta-Learning. find
  20. Tsendsuren Munkhdalai and Hong Yu. 2017. Meta networks. arXiv preprint arXiv:1703.00837. — [arXiv:1703.00837](https://arxiv.org/abs/1703.00837) find
  21. Keerthiram Murugesan, Jaime Carbonell, and Yiming Yang. 2017. Co-clustering for multitask learning. arXiv preprint arXiv:1703.00994. — [arXiv:1703.00994](https://arxiv.org/abs/1703.00994) find
  22. Andrew Y Ng, Michael I Jordan, and Yair Weiss. 2002. On spectral clustering: Analysis and an algorithm. In Advances in neural information processing systems, pages 849–856. find
  23. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532– 1543. our card
  24. Sachin Ravi and Hugo Larochelle. 2017. Optimization as a model for few-shot learning. In International Conference on Learning Representations, volume 1, page 6. find
  25. Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175. find
  26. Eleni Triantafillou, Richard Zemel, and Raquel Urtasun. 2017. Few-shot learning through an information retrieval lens. In Advances in Neural Information Processing Systems, pages 2252–2262. find
  27. Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pages 3630–3638. find
  28. Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. 2017. Learning to model the tail. In Advances in Neural Information Processing Systems 30, pages 7032–7042. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2018-diverse-few-shot-text-classification/