Elamur Research⌕
☰
home / papers / 2017 / Adversarial Multi-task Learning for Text Classification

Adversarial Multi-task Learning for Text Classification

Summary

The paper tackles a weakness of neural multi-task learning for text classification. In the usual shared-private setup, the "shared" representation can absorb features that matter only to one task, and genuinely common features can end up in the private layers. The authors build an LSTM-based shared-private architecture, ASP-MTL, and add two mechanisms to keep the spaces clean:

  • Adversarial training: a task discriminator tries to guess which task a sentence came from using only the shared encoding, and the shared LSTM learns to defeat it.
  • Orthogonality constraint: this pushes shared and private features apart, so the two spaces do not duplicate each other.

The authors generalise the adversarial objective from two classes to many, so all tasks can be trained jointly. Because the task-identity signal needs no class labels, unlabeled text can also be used. Experiments cover 16 text classification tasks, whose datasets are publicly released. ASP-MTL outperforms the plain shared-private baseline. The learned shared layer also works as a reusable, off-the-shelf component for new tasks. This matters for any setting with many small related datasets.

Key points

  • The authors compare three architectures: FS-MTL (one LSTM shared by all tasks), SP-MTL (a shared LSTM plus a private LSTM per task, with their outputs concatenated), and the proposed ASP-MTL.
  • The total loss combines three terms: weighted per-task cross-entropy, a multi-class adversarial loss L_Adv over task identity, and a difference/orthogonality loss L_Diff.
  • The evaluation uses 16 text classification datasets, mostly sentiment over product and movie review domains. They are available at nlp.fudan.edu.cn/data/.
  • In the transfer experiments, the shared layer is frozen knowledge applied to an unseen task. Two transfer strategies were tested, and the Bi-Channel variant, which also has a task-specific layer, did better.
  • Qualitative analysis (Table 4, neuron heat maps) found that in SP-MTL, task-specific cues such as "infantile" leak into the shared layer. In ASP-MTL, shared and private layers overlap only slightly in what they capture.

Interesting ideas

  • I1. A discriminator that guesses the source task: using such a discriminator as an adversary is a cheap way to force a representation to be source-agnostic. The same trick could apply to mixing Ukrainian corpora from different domains or genres, or to removing dataset artefacts.
  • I2. Explicit separation through orthogonality: penalising overlap between shared and private features keeps them from duplicating each other. This makes the capacity of each part more meaningful and is easy to add to any multi-branch model.
  • I3. Unlabeled data for free: the adversarial term only needs to know which task or corpus a text belongs to, not its class label. Raw in-domain text can therefore improve the shared encoder, which is useful for low-resource languages.
  • I4. Training a reusable shared layer once: a shared encoder learned across many tasks can be plugged into a new task, ideally with an added private channel. This gives a lightweight form of transfer learning with no huge pretraining.
  • I5. An interpretability check for sharing schemes: the authors list which words activate shared versus private neurons, using a word like "infantile" whose sentiment differs by domain. This is a simple diagnostic for whether the sharing behaves as intended.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

122 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1001

@inproceedings{liu2017adversarial,
  title = {{Adversarial Multi-task Learning for Text Classification}},
  author = {Liu, Pengfei and Qiu, Xipeng and Huang, Xuanjing},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1001},
  url = {https://doi.org/10.18653/v1/p17-1001}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find
  2. Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. 2014. Domain-adversarial neural networks. arXiv preprint arXiv:1412.4446 . — [arXiv:1412.4446](https://arxiv.org/abs/1412.4446) find
  3. Yangqing Jia, Mathieu Salzmann, and Trevor Darrell. 2010. Factorized latent spaces with structured sparsity. In Advances in Neural Information Processing Systems. pages 982–990. find
  4. Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine learning 79(1-2):151–175. find
  5. Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In Proceedings of The 32nd International Conference on Machine Learning. find
  6. Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. 2007. Analysis of representations for domain adaptation. Advances in neural information processing systems 19:137. find
  7. John Blitzer, Mark Dredze, Fernando Pereira, et al. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In ACL. volume 7, pages 440–447. find our card
  8. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of ACL. find our card
  9. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130 . find our card
  10. Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016. Domain separation networks. In Advances in Neural Information Processing Systems. pages 343– 351. find
  11. Pengfe Liu, Xipeng Qiu, Jifan Chen, and Xuanjing Huang. 2016a. Deep fusion LSTMs for text semantic matching. In Proceedings of ACL. find
  12. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 . find
  13. PengFei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuanjing Huang. 2015a. Multi-timescale long short-term memory neural network for modelling sentences and documents. In Proceedings of the Conference on EMNLP. find
  14. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML. find
  15. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016b. Deep multi-task learning with shared memory. In Proceedings of EMNLP. find
  16. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The JMLR 12:2493–2537. find our card
  17. PengFei Liu, Xipeng Qiu, and Xuanjing Huang. 2016c. Recurrent neural network for text classification with multi-task learning. In Proceedings of International Joint Conference on Artificial Intelligence. find
  18. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science 14(2):179–211. find
  19. Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. 2015b. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In NAACL. find our card
  20. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114 . find our card
  21. Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the ACL. pages 142–150. find our card
  22. Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 3994–4003. find
  23. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 115–124. find our card
  24. Gwangbeen Park and Woobin Im. 2016. Image-text multi-modal representation learning by adversarial backpropagation. arXiv preprint arXiv:1612.08354 . — [arXiv:1612.08354](https://arxiv.org/abs/1612.08354) find
  25. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the EMNLP 12:1532– 1543. find
  26. Mathieu Salzmann, Carl Henrik Ek, Raquel Urtasun, and Trevor Darrell. 2010. Factorized orthogonal latent spaces. In AISTATS. pages 701–708. find
  27. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP. find
  28. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in NIPS. pages 3104–3112. find
  29. Yaniv Taigman, Adam Polyak, and Lior Wolf. 2016. Unsupervised cross-domain image generation. arXiv preprint arXiv:1611.02200 . — [arXiv:1611.02200](https://arxiv.org/abs/1611.02200) find
  30. Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. 2014. Facial landmark detection by deep multi-task learning. In European Conference on Computer Vision. Springer, pages 94–108. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2017-adversarial-multi-task-learning-text/