Elamur Research⌕
☰
home / papers / 2020 / Simple Data Augmentation with the Mask Token Improves Domain Adaptation for Dialog Act Tagging

Simple Data Augmentation with the Mask Token Improves Domain Adaptation for Dialog Act Tagging

Paper · 2020 · Semih Yavuz, Kazuma Hashimoto, Wenhao Liu et al.

Summary

Dialog act (DA) taggers learn speaker intentions such as "request" or "inform". These intentions are shared across task-oriented domains, yet taggers trained on one domain transfer poorly to another, and labeling new domains is costly. The authors adapt DA taggers using only unlabeled target-domain dialogues. Their method, MaskAugment, perturbs input by randomly swapping tokens for BERT's pre-trained MASK token at a chosen rate.

They build two objectives on top of it:

  • Masked tagging loss (MTL): supervised tagging on masked input.
  • Disagreement loss (DAL): a teacher-student consistency loss. The same unlabeled turn is masked twice. The lightly masked copy acts as teacher, and its predictions become soft targets for the heavily masked student copy.

The tagger is multi-label BERT over flattened dialogue history with speaker tags. Experiments on GSim (movie ↔ restaurant) and SGD (music, media, ride-sharing → flights) show that BERT already beats an earlier LSTM baseline by a wide margin. MaskAugment adds up to about 3 F1 points with full source data and up to about 10 points in low-resource settings. Combined with domain-adaptive pre-training and MLM fine-tuning, target F1 reaches 94.4 on GRes.

Key points

  • Task: turn-level multi-label classification with one sigmoid per act. Input is [CLS] plus history, with [USR]/[SYS] speaker tokens. Segment id 0 marks past turns and 1 marks the current turn. Max length is 128 tokens, about 10 turns.

  • Data:

    • GSim has 1500/469/1117 dialogues, mapped to 13 universal-schema tags.
    • SGD has 22,825 dialogues; small single domains serve as sources and SFlights as target.

  • GMov → GRes results:

    • LSTM (Paul et al., 2019): 75.1 target micro-F1.
    • scratch-BERT, STL only: 89.7.
    • STL+MTL+DAL: 92.8.
    • Adding MLM on target text: 94.1 (scratch-BERT) and 94.4 (pre-BERT, domain-adaptively pre-trained).

  • Low-resource setting (labeled source dialogues only):

    Model10 dialogues, STL10 dialogues, MaskAugment50 dialogues, MaskAugment
    scratch-BERT53.358.4—
    pre-BERT59.870.485.1

  • Training details:

    • Teacher mask rate ε_t ∈ [0, 0.1]; student rate ε_s ∈ [0.1, 0.5].
    • DAL is switched on only after one epoch.
    • MLM loss is weighted 0.1.
    • Gains come on both precision and recall.
    • Some rare acts (sys-negate, sys-notify-failure) get worse under scratch-BERT.

Interesting ideas

  • I1. Two masking strengths as teacher and student: the less-masked view keeps more of the original and so serves as a more trustworthy teacher for the more-masked view, which turns unlabeled text into a training signal without any external augmenter.
  • I2. Masking as an augmentation that needs no generation: back-translation and word swaps are fragile for multi-turn text, while masking is cheap, controllable by one rate, and native to the pretrained model's vocabulary.
  • I3. Delayed consistency loss: the unsupervised disagreement term starts only after an epoch of supervised training, so the teacher is not random noise at first.
  • I4. Speaker tags plus segment ids for current vs. past turns: a lightweight way to encode dialogue history so a classifier attends to the current utterance while still seeing context.
  • I5. Per-label breakdown weighted by frequency: the paper reports F1 per act alongside its frequency and shows that average gains can hide drops on rare acts.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): unlabeled English sentences could train a tagger or parser for consistency between a lightly and a heavily masked copy, adding to silver data from witness parsers.
  • I2 (tree): gparse could be trained with token-masked inputs so attachment decisions lean on structure rather than specific domain words, which may help when moving between treebank genres.
  • I3 (learn): any self-training or self-play loop should first warm up on gold or agreed silver data before trusting the model's own soft targets.
  • I4 (frames): the speech-act component in prag could mark the current utterance versus prior turns this way when tagging requests, offers, or confirmations in dialogue text.
  • I5 (answer): evaluation reports for readers and students should list per-category scores with frequencies, so regressions on rare question types are not masked by overall accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢SalesforceSan Francisco · United States37 works · 15,019 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Semih Yavuz, Kazuma Hashimoto, Wenhao Liu, Nitish Shirish Keskar, Richard Socher and Caiming Xiong. 2020. Simple Data Augmentation with the Mask Token Improves Domain Adaptation for Dialog Act Tagging. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.412

@inproceedings{yavuz2020simple,
  title = {{Simple Data Augmentation with the Mask Token Improves Domain Adaptation for Dialog Act Tagging}},
  author = {Semih Yavuz, Kazuma Hashimoto, Wenhao Liu, Nitish Shirish Keskar, Richard Socher and Caiming Xiong},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.412},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.412}
}

References

34 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020. PLATO: Pre-trained dialogue generation model with discrete latent variable. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
  2. David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. 2020. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In International Conference on Learning Representations (ICLR). ↗
  3. David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. 2019. Mixmatch: A holistic approach to semi-supervised learning. In Conference on Neural Information Processing Systems (NeurIPS). ↗
  4. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find
  5. Harry Bunt, Jan Alexandersson, Jean Carletta, Alex Chengyu Fang, Koiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, Claudia Soria, and David Traum. 2010. Towards an ISO standard for dialogue act annotation. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10).
  6. Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). ↗
  7. Zheqian Chen, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. 2018. Dialogue act recognition via crf-attentive structured network. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR).
  8. Mark G Core and James F Allen. 1997. Coding dialogs with the damsl annotation scheme. In AAAI Conference on Artificial Intelligence (AAAI). find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  10. John J. Godfrey, Edward Holliman, and Jan McDaniel. 1992. Switchboard: telephone speech corpus for research and development. [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing. find
  11. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. find
  12. Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL). find
  13. Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019. Training neural response selection for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. ↗
  14. Dan Jurafsky, Liz Shriberg, and Debra Biasca. 1997. Switchboard swbd-damsl shallow-discoursefunction annotation coders manual, draft 13. find
  15. Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, and Sachindra Joshi. 2018. Dialogue act sequence labeling using hierarchical encoder with crf. In AAAI Conference on Artificial Intelligence (AAAI). ↗
  16. Samuli Laine and Timo Aila. 2017. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations (ICLR). ↗
  17. Junnan Li, Richard Socher, and Steven C. H. Hoi. 2020. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations (ICLR). ↗
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. ↗
  19. Ilya Loshchilov and Frank Hutter. 2017. weight decay regularization in adam. abs/1711.05101. ↗
  20. Fixing ArXiv, Stefano Mezza, Alessandra Cervone, Evgeny Stepanov, Giuliano Tortoreto, and Giuseppe Riccardi. 2018. ISO-standard domain-independent dialogue act tagging for conversational agents. In Proceedings of the 27th International Conference on Computational Linguistics. ↗
  21. Shachi Paul, Rahul Goel, and Dilek Z. Hakkani-Tür. 2019. Towards universal dialogue act tagging for task-oriented dialogues. In INTERSPEECH.
  22. Vipul Raheja and Joel Tetreault. 2019. Dialogue Act Classification with Context-Aware Self-Attention. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ↗
  23. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In AAAI Conference on Artificial Intelligence (AAAI). ↗
  24. Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation for consistency training. arXiv: Learning.
  25. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B. Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. ArXiv, abs/1911.00536. ↗
  26. Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tas-dizen. 2016. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Conference on Neural Information Processing Systems (NeurIPS). find
  27. Pararth Shah, Dilek Z. Hakkani-Tür, Gökhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak Kennard, and Larry Heck. 2018. Building a conversational agent overnight with dialogue self-play. ArXiv, abs/1801.04871. ↗
  28. Sam Shleifer. 2019. Low resource text classification with ulmfit and backtranslation. CoRR. ↗
  29. Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. 2020. Fix-match: Simplifying semi-supervised learning with consistency and confidence. ArXiv. find
  30. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics. ↗
  31. Ashish Vaswani, Shazeer Noam, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Conference on Neural Information Processing Systems (NeurIPS).
  32. Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics. ↗
  33. Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. ArXiv, abs/1901.08149. ↗
  34. Chien-Sheng Wu, Steven C. H. Hoi, Richard Socher, and Caiming Xiong. 2020. Tod-bert: Pre-trained natural language understanding for task-oriented dialogues. ArXiv, abs/2004.06871. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yavuz-2020-simple-data-augmentation-mask-token/