Elamur Research⌕
☰
home / papers / 2019 / Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

Paper · 2019 · Zi-Yi Dou, Keyi Yu and Antonios Anastasopoulos

Summary

General-purpose text representations such as BERT and multi-task models like MT-DNN often underperform when a target task has little labeled data. Multi-task training also tends to favor the tasks with the most data. The authors test whether optimization-based meta-learning gives a better starting point for low-resource English NLU tasks. They compare MAML, first-order MAML and Reptile. The pipeline has three stages: BERT BASE pre-training, a meta-learning phase on four data-rich GLUE tasks (SST-2, QQP, MNLI, QNLI), and fine-tuning on four small GLUE tasks (CoLA, MRPC, STS-B, RTE). WNLI is excluded, and BERT and MT-DNN serve as baselines. On the GLUE test sets, the meta-learned models generally beat both baselines, and Reptile is the strongest overall. Ablations cover how tasks are sampled and the settings of the inner update loop. A transfer experiment on the unseen SciTail dataset shows the largest benefit when very little training data is available. This suggests meta-learning is a practical way to initialize models for data-scarce NLU settings.

Key points

  • GLUE test results for Reptile vs. MT-DNN: CoLA 53.2 vs 51.7, MRPC 90.2/86.7 vs 89.9/86.3, STS-B 88.7/88.1 vs 87.6/86.8, RTE 77.0 vs 75.4. BERT reaches only 66.4 on RTE. Vanilla MAML scores best on CoLA (53.4).
  • Task sampling: sampling auxiliary tasks in proportion to dataset size (PPS) beat uniform and mixed sampling. On RTE dev, Reptile scored 83.0 with PPS and 75.7 with uniform sampling.
  • Meta-learning setup: 5 inner update steps, inner learning rate α = 1e-3 and 8 tasks per step. Training used Adam with batch size 32 and learning rate 5e-5 for 5 epochs.
  • Inner-loop ablations: both fewer and more inner steps hurt, since k = 1 reduces Reptile to plain joint training. A larger α also degraded results.
  • Transfer to SciTail: with 0.1%, 1%, 10% and 100% of the training data, Reptile consistently outperformed MT-DNN. Its gains over BERT were largest in the smallest-data settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Low-resource languages · 166 works

Cite

Cite as Zi-Yi Dou, Keyi Yu and Antonios Anastasopoulos. 2019. Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1112

@inproceedings{dou2019investigating,
  title = {{Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks}},
  author = {Zi-Yi Dou, Keyi Yu and Antonios Anastasopoulos},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1112},
  url = {https://doi.org/10.18653/v1/d19-1112}
}

References

35 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Cera, Mona Diabb, Eneko Agirrec, Inigo Lopez-Gazpioc, Lucia Speciad, and Basque Country Donostia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and cross-lingual focused evaluation. In 11th International Workshop on Semantic Evaluations. our card
  2. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JMLR. our card
  3. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. find
  4. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NeurIPS. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. our card
  6. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). our card
  7. Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, and Tie-Yan Liu. 2018. Learning to teach. In ICLR. find
  8. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML. find
  9. Jiatao Gu, Yong Wang, Yun Chen, Victor OK Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In EMNLP. our card
  10. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In CVPR. find
  11. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. find
  12. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI. find
  13. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  14. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In NeurIPS. find
  15. Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science. find
  16. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In ICML. our card
  17. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  18. Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. 2015. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In NAACL. our card
  19. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. arXiv. our card
  20. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. arXiv. our card
  21. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NeurIPS. our card
  22. Kun Qian and Zhou Yu. 2019. Domain adaptive dialog generation via meta learning. In ACL. find
  23. Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. find
  24. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. our card
  25. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… find
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS. our card
  27. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR. our card
  28. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471. find
  29. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL. find
  30. Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. 2018. Deep layer aggregation. In CVPR. find
  31. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NeurIPS. our card
  32. Alex Nichol, Joshua Achiam, and John Schulman. 2018. On first-order meta-learning algorithms. arXiv. find
  33. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP. our card
  34. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. our card
  35. Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohamadi, and Sanjeev Khudanpur. 2018. Semi-orthogonal low-rank matrix factorization for deep neural networks. In Inter-Speech. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dou-2019-investigating-meta-learning-algorithms-low/