Elamur Research⌕
☰
home / papers / 2022 / Improving Dialogue Act Recognition with Augmented Data

Improving Dialogue Act Recognition with Augmented Data

Paper · 2022 · Khyati Mahajan, Soham Parikh, Quaizar Vohra et al.

Summary

The authors build dialogue act recognition (DAR) models for an industrial virtual assistant. They find that a BERT-based classifier fine-tuned on the Schema-Guided Dialogue (SGD) corpus scores very well on SGD but fails on in-house conversations. AFFIRM falls to 0.39 F1 and NEGATE to 0.16. The cause is low variety in SGD: nearly every affirmation contains "yes" or a close synonym. Real users, by contrast, often confirm or refuse implicitly by echoing words from the bot's question. To fix this, the team used GPT-3 to generate affirming and negating replies. They tried few-shot and zero-shot prompts and varied how much conversation context each prompt included. They then studied oversampling of the synthetic set, mixing in linguist-written data, and skipping human filtering. Adding linguistically varied synthetic data raised the scores to 0.85 (AFFIRM) and 0.88 (NEGATE). This shows that cheap LLM generation can close a domain gap that a large but stylistically narrow corpus leaves open.

Key points

  • Data sizes: SGD has 16,142 dialogues and 164,982 utterances (25,054 AFFIRM, 16,715 NEGATE). The synthetic internal training set has 161 dialogues and 1,980 utterances. The linguist-written test set has 296 dialogues and 1,629 utterances.
  • When the synthetic set is duplicated k times, F1 rises up to a point and then drops again. This shows that the balance between SGD and synthetic data matters.
  • In the linguist-data experiment, 20% of the SME data is used for training and 80% for evaluation. SME data alone beats synthetic data alone, especially for NEGATE. Combining both works best, reaching about 0.88–0.89 F1.
  • Unfiltered output from the two best prompts (about 1k utterances) matched or beat human-vetted output pooled from all prompts (about 1.9k). The best result was 0.85/0.88 with Synthetic-all-noisy.
  • Practical prompt lessons for GPT-3:
    • end questions with a question mark;
    • mark speakers with a "Bot:" prefix instead of describing who asked;
    • give several short instructions rather than one long one.

Interesting ideas

  • I1. Look for lexical shortcuts in the training data before blaming the model: a label that almost always co-occurs with one trigger word (here "yes") yields a classifier that cannot handle paraphrases or implicit acts.
  • I2. Ask the generator to answer using only words from the question, which produces implicit replies that confirm or refuse by reusing the prompt's content instead of a polarity word.
  • I3. Sweep the oversampling factor of a small augmented set against a big base corpus, and pick the peak rather than assuming more duplication is better.
  • I4. Prompt quality can matter more than filtering: unvetted output from a couple of well-chosen prompts outperformed carefully filtered output from many prompts.
  • I5. Evaluate on a test set written by linguists to mimic real users, kept separate from the generated training data, so that domain-transfer failures become visible.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Audit the speech-act rules in prag for over-reliance on cue words like "yes/no", and add tests where assent or refusal is expressed only by echoing the question's verb ("I'd like to", "skip").
  • I2 (learn): Use this kind of instruction when an LLM teacher creates silver examples of indirect answers, so speech-act detection is trained on content-echo patterns rather than particles.
  • I3 (learn): When mixing small synthetic or self-play sets into treebank training, run a duplication-factor sweep with held-out evaluation to find the mixing ratio where gains stop.
  • I4 (learn): Invest review effort in selecting and validating a few teacher prompts once, then accept their output with lighter checks, saving human time in the LLM teacher tiers.
  • I5 (answer): Keep hand-written, style-shifted test sets for the school-test and fable QA readers so that overfitting to the style of generated training text is caught early.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢ServiceNowSanta Clara · United States5 works · 849 citations

Also on the byline: University of North Carolina at Charlotte.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Dialogue systems · 401 works Speech act theory · 160 works

Cite

Cite as Khyati Mahajan, Soham Parikh, Quaizar Vohra, Mitul Tiwari and Samira Shaikh. 2022. Improving Dialogue Act Recognition with Augmented Data. Proceedings of the Second Workshop on Natural Language Generation, Evaluation, and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.gem-1.44

@inproceedings{mahajan2022improving,
  title = {{Improving Dialogue Act Recognition with Augmented Data}},
  author = {Khyati Mahajan, Soham Parikh, Quaizar Vohra, Mitul Tiwari and Samira Shaikh},
  booktitle = {Proceedings of the Second Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)},
  year = {2022},
  doi = {10.18653/v1/2022.gem-1.44},
  url = {https://doi.org/10.18653/v1/2022.gem-1.44}
}

References

23 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ali Ahmadvand, Jason Ingyu Choi, and Eugene Agichtein. 2019. Contextual dialogue act classification for open-domain conversational agents. In Proceedings of the 42nd international acm sigir conference on research and development in information retrieval, pages 1273–1276. our card
  2. Vevake Balaraman, Seyedmostafa Sheikhalishahi, and Bernardo Magnini. 2021. Recent neural methods on dialogue state tracking for task-oriented dialogue systems: A survey. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 239–251. find
  3. Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021. find
  4. Improving question answering model robustness with synthetic adversarial data generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8830–8848. American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3727–3733. find
  5. Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. Inpars: Data augmentation for information retrieval using large language models. arXiv preprint arXiv:2202.05144. ↗
  6. Kristy Boyer, Eun Young Ha, Robert Phillips, Michael Wallis, Mladen Vouk, and James Lester. 2010. Dialogue act modeling in a complex task-oriented domain. In Proceedings of the SIGDIAL 2010 Conference, pages 297–305. our card
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. our card
  8. Zheqian Chen, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. 2018. Dialogue act recognition via crf-attentive structured network. In The 41st international acm sigir conference on research & development in information retrieval, pages 225–234. ↗ our card
  9. Pierre Colombo, Emile Chapuis, Matteo Manica, Emmanuel Vignon, Giovanna Varni, and Chloe Clavel. 2020. Guiding attention in sequence-to-sequence models for dialogue act prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7594–7601. ↗ our card
  10. Sergio Grau, Emilio Sanchis, Maria Jose Castro, and David Vilar. 2004. Dialogue act classification using a bayesian approach. In 9th Conference Speech and Computer. find
  11. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Schema-guided dialogue state tracking task at dstc8. arXiv preprint arXiv:2002.01359. ↗
  12. Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7. our card
  13. Tulika Saha, Saurabh Srivastava, Mauajama Firdaus, Sriparna Saha, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Exploring machine learning and deep learning frameworks for task-oriented dialogue act classification. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. find
  14. Pararth Shah, Dilek Hakkani-Tür, Gokhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018. Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871. ↗
  15. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235. our card
  16. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗ our card
  17. Maryam Tavafi, Yashar Mehdad, Shafiq Joty, Giuseppe Carenini, and Raymond Ng. 2013. Dialogue act recognition in synchronous and asynchronous conversations. In Proceedings of the SIGDIAL 2013 Conference, pages 117–121. ↗ our card
  18. Han Guo, Bowen Tan, Zhengzhong Liu, Eric P Xing, and Zhiting Hu. 2021. Text generation with efficient (soft) q-learning. arXiv preprint arXiv:2106.07704. ↗
  19. Léo Jacqmin, Lina M Rojas-Barahona, and Benoit Favre. 2022. " do you follow me?": A survey of recent approaches in dialogue state tracking. arXiv preprint arXiv:2207.14627. ↗
  20. Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, and Sachindra Joshi. 2018. Dialogue act sequence labeling using hierarchical encoder with crf. In Proceedings of the aaai conference on artificial intelligence, volume 32. ↗ our card
  21. Dewayne Whitfield. 2021. Using gpt-2 to create synthetic data to improve the prediction performance of nlp machine learning classification models. arXiv preprint arXiv:2104.10658. ↗
  22. Bill Noble and Vladislav Maraev. 2021. Large-scale text pre-training helps with dialogue act recognition, but not without fine-tuning. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 166–172. our card
  23. Vipul Raheja and Joel Tetreault. 2019. Dialogue act classification with context-aware self-attention. In Proceedings of the 2019 Conference of the North ↗ our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mahajan-2022-improving-dialogue-act-recognition-augmented/