Elamur Research⌕
☰
home / papers / 2020 / Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network

Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network

Summary

The paper addresses slot tagging for task-oriented dialogue when a new domain provides only one or five labeled support sentences per label. Treating each token as a separate few-shot classification ignores dependencies between labels. However, transition statistics cannot be reused directly, because every domain has its own label set. The authors build a few-shot CRF. Its emission scores come from word-label similarity, and its transition scores come from collapsed dependency transfer (CDT). CDT maps domain-specific tags onto abstract O/B/I categories, separating same-type from different-type transitions, then expands the learned abstract table into a full matrix for any unseen domain. For emissions they propose L-TapNet, which extends TapNet by building its projection space from label-name semantics and prototypes. It also uses pair-wise embeddings that encode a query together with its support context. The model is trained on source domains and tested without fine-tuning on slot tagging (SNIPS) and NER. In the one-shot setting it beats the strongest few-shot baseline by 14.64 F1, showing that label dependencies and label names are useful signals when annotations are scarce.

Key points

  • CDT learns a compact 3×5 table of abstract transitions (from O/B/I into O, same-type B, different-type B, same-type I, different-type I). Start and End are added as extra abstract labels, and the table is expanded to fit each target label set.
  • Replacing CDT with greedy transition rules is clearly worse. L-TapNet alone scores 60.40 (1-shot) and 66.93 (5-shot) F1, L-TapNet+Rule scores 65.30 and 69.64, and L-TapNet+CDT scores 70.41 and 75.01.
  • In ablations, removing CDT causes large F1 drops in every setting. Dropping the Begin/Inner words from label representations costs 0.97 F1 on 1-shot SNIPS. Removing prototypes hurts most in the 5-shot setting.
  • Bi-gram analysis on 1-shot slot tagging shows CDT helps at span borders as well as inside spans. For example, I-B/B-B accuracy rises from 65.0% to 72.5% and I-O from 76.7% to 81.7%.
  • Code is released at github.com/AtmaHou/FewShotTagging.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

83 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.128

@inproceedings{hou2020few,
  title = {{Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network}},
  author = {Hou, Yutai and Che, Wanxiang and Lai, Yongkui and Zhou, Zhihan and Liu, Yijia and Liu, Han and Liu, Ting},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.128},
  url = {https://doi.org/10.18653/v1/2020.acl-main.128}
}

References

37 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2017. Towards zero-shot frame semantic parsing for domain scaling. arXiv preprint arXiv:1707.02363. — [arXiv:1707.02363](https://arxiv.org/abs/1707.02363) find
  2. Yun-Nung Chen, Dilek Hakkani-Tür, and Xiaodong He. 2016. Zero-shot learning of intent embeddings for expansion by convolutional deep structured semantic models. In Proc. of the ICASSP, pages 6045– 6049. IEEE. find
  3. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-desig… find our card
  4. Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proc. of the 3rd Workshop on Noisy User-generated Text, pages 140–147. find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. of the NAACL-HLT, Volume 1 (Long and Short Papers), pages 4171–4186. find
  6. Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In Proc. of the ACL. find our card
  7. Li Fei-Fei. 2006. Knowledge transfer in learning to recognize visual objects classes. In International Conference on Development and Learning, pages 1–8. find
  8. Li Fei-Fei, Rob Fergus, and Pietro Perona. 2006. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594–611. find
  9. Michael Fink. 2005. Object classification from a single example utilizing class relevance metrics. In NIPS, pages 449–456. find
  10. G David Forney. 1973. The viterbi algorithm. Proc. of the IEEE, 61(3):268–278. find
  11. Alexander Fritzler, Varvara Logacheva, and Maksim Kretov. 2019. Few-shot classification in named entity recognition task. In Proc. of the SAC, pages 993– 1000. find
  12. Tianyu Gao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2019. Neural snowball for few-shot relation learning. arXiv preprint arXiv:1908.11007. — [arXiv:1908.11007](https://arxiv.org/abs/1908.11007) find
  13. Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. In Proc. of the EMNLP-IJCNLP, pages 3895–3904. find
  14. Yutai Hou, Yijia Liu, Wanxiang Che, and Ting Liu. 2018. Sequence-to-sequence data augmentation for dialogue language understanding. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 1234–1245. find
  15. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proc. of the ACL, Volume 1: Long Papers, pages 328–339. find
  16. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. find our card
  17. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of the ICLR. find
  18. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. of the ICML, ICML ’01, pages 282–289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  19. Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338. find
  20. Sungjin Lee and Rahul Jha. 2019. Zero-shot adaptive transfer for conversational language understanding. In Proc. of the AAAI, volume 33, pages 6642–6649. find
  21. Bingfeng Luo, Yansong Feng, Zheng Wang, Songfang Huang, Rui Yan, and Dongyan Zhao. 2018. Marrying up regular expressions with neural networks: A case study for spoken language understanding. In Proc. of the ACL, Volume 1: Long Papers, pages 2083–2093. find
  22. Xin Lv, Yuxian Gu, Xu Han, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2019. Adapting meta knowledge graph information for multi-hop reasoning over few-shot relations. arXiv preprint arXiv:1908.11513. find
  23. Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In Proc. of the ACL, Volume 1: Long Papers. find our card
  24. Erik G Miller, Nicholas E Matsakis, and Paul A Viola. 2000. Learning from one example through shared densities on transforms. In Proc. of the CVPR, volume 1, pages 464–471. IEEE. find
  25. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proc. of the EMNLP, pages 1532– 1543. find
  26. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In Proc. of the CoNLL, pages 143–152. find
  27. Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In Proc. of the EMNLP. find our card
  28. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proc. of the CoNLL-HLT-NAACL, pages 142–147. find
  29. Ruhi Sarikaya, Paul A Crook, Alex Marin, Minwoo Jeong, Jean-Philippe Robichaud, Asli Celikyilmaz, Young-Bum Kim, Alexandre Rochette, Omar Zia Khan, Xiaohu Liu, et al. 2016. An overview of end-to-end language understanding and dialog management for personal digital assistants. In 2016 IEEE Spoken Lan… find
  30. Mike Schuster and Kuldip K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Trans. Signal Processing, 45(11):2673–2681. find
  31. Ting Liu, Yiming Cui, Qingyu Yin, Wei-Nan Zhang, Shijin Wang, and Guoping Hu. 2017. Generating and exploiting large-scale pseudo training data for zero pronoun resolution. In Proc. of the ACL, pages 102–111. find our card
  32. Eli Schwartz, Leonid Karlinsky, Rogerio Feris, Raja Giryes, and Alex M Bronstein. 2019. Baby steps towards few-shot learning with multiple semantics. arXiv preprint arXiv:1906.01905. — [arXiv:1906.01905](https://arxiv.org/abs/1906.01905) find
  33. Zhiwu Lu, Jiechao Guan, Aoxue Li, Tao Xiang, An Zhao, and Ji-Rong Wen. 2018. Zero and few shot learning with semantic feature synthesis and competitive learning. arXiv preprint arXiv:1810.08332. — [arXiv:1810.08332](https://arxiv.org/abs/1810.08332) find
  34. Darsh J. Shah, Raghav Gupta, Amir A. Fayazi, and Dilek Hakkani-Tür. 2019. Robust zero-shot cross-domain slot filling with example values. In Proc. of the ACL, Volume 1: Long Papers, pages 5484–5490. find
  35. Y. Shin, K. M. Yoo, and S. Lee. 2019. Utterance generation with variational auto-encoder for slot filling in spoken language understanding. IEEE Signal Processing Letters, 26(3):505–509. find
  36. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proc. of the NIPS, pages 649–657. find our card
  37. Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In NIPS, pages 4077–4087. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hou-2020-few-shot-slot-tagging-collapsed/