Elamur Research⌕
☰
home / papers / 2024 / Dual Process Masking for Dialogue Act Recognition

Dual Process Masking for Dialogue Act Recognition

Paper · 2024 · Yeo Jin Kim, Halim Acosta, Wookhee Min et al.

Summary

The paper addresses dialogue act recognition: labelling each utterance in a conversation with its communicative function. It focuses on small, domain-specific corpora with their own label sets, where labelled data is scarce. The authors propose Dual-Process Masking (DP-Masking), loosely modelled on dual-process theories of cognition, and it runs in two stages. In the "reflective" stage, a masker and a classifier hide tokens and search for masks that still lead to the correct label. Masks from correctly classified utterances are kept in a working memory, and fresh random masks are added each round for exploration. In the "intuitive" stage, a second masker learns only from those stored masks, and a classifier is trained on the resulting masked input. The method was tested on a collaborative problem-solving (CPS) corpus of about 2K labelled items and on three public benchmarks, with Llama-2-7B, Llama-3-8B and Flan-T5-small as backbones. It gives clear gains when data is scarce and stays competitive on large datasets. The surviving masks also show which tokens carry each act.

Key points

  • On the CPS data, DP-Masking reaches state-of-the-art results. In macro F1, its average normalized improvement is 13.4% over random masking and 17.6% over no masking.
  • The public benchmarks are Oasis (15K), TalkMoves (174K) and SwDA (199K). Across growing data sizes, DP-Masking beats random masking by 3.2% and no masking by 8.3% in MCC (average normalized improvement).
  • With full data, DP-FT5 sets a new best of 70.8% accuracy on Oasis, surpassing the 69.4% previously reported by Chapuis et al. (2020). Every FT5 variant achieves 81% MCC on TalkMoves, compared with the previous 78.1%. On SwDA, the method gives no significant gain.
  • The benefit is largest on 5% data subsets and nearly vanishes with full data. The authors read this as masking substituting for extra training data.
  • Stated limitations: the masker ignores correlations among masked tokens, experiments cover English only, and the approach is GPU-heavy. Code is at github.com/ykim32/DPMasking.

Interesting ideas

  • I1. Keep only the masks that still produced a correct prediction, store them across iterations, and add random masks each round to explore; this finds a minimal set of cue tokens for a label without hand annotation.
  • I2. Separate a slow search phase that discovers good supervision from a fast model that is trained only on what the search confirmed, so inference stays cheap while training does the deliberation.
  • I3. Use random masking at the same rate as a matched control, so that any gain can be credited to choosing which tokens to hide rather than to masking itself.
  • I4. Report results on several training-set fractions (e.g., 5% vs full), not only on full data, to show in which data regime a technique actually pays off.
  • I5. Inspect which tokens survive masking for each class to refine label taxonomies and annotation guidelines.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The speech-act module in prag could use this search to find which words in an utterance actually trigger "request" vs "promise", yielding an explicit, inspectable cue list per act.
  • I2 (learn): Witness-parser agreement or LLM-teacher checks could act as the slow phase, with the fast perceptron or gparse model trained only on the confirmed, filtered silver output.
  • I3 (learn): This fits Mova's negative-control evaluation: every learned filter or repair rule should be compared against a random variant with the same budget.
  • I4 (learn): Mova could add data-size curves when testing new silver-data or teacher tiers, which would show whether a trick helps only in low-resource settings such as new frame types.
  • I5 (frames): Comparing survivor tokens across speech-act or irony labels could expose overlapping categories in prag and suggest clearer, explainable definitions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North Carolina at Chapel HillChapel Hill · United States67 works · 16,474 citations🎓North Carolina State UniversityRaleigh · United States13 works · 406 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSpeech act theory · 229 works

Cite

Cite as Yeo Jin Kim, Halim Acosta, Wookhee Min, Jonathan Rowe, Bradford Mott, Snigdha Chaturvedi and James Lester. 2024. Dual Process Masking for Dialogue Act Recognition. Findings of the Association for Computational Linguistics: EMNLP 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-emnlp.895

@inproceedings{kim2024dual,
  title = {{Dual Process Masking for Dialogue Act Recognition}},
  author = {Yeo Jin Kim, Halim Acosta, Wookhee Min, Jonathan Rowe, Bradford Mott, Snigdha Chaturvedi and James Lester},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.895},
  url = {https://doi.org/10.18653/v1/2024.findings-emnlp.895}
}

References

47 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Ahmadvand, J. Ingyu Choi, and E. A. Agichtein. 2019. Contextual dialogue act classification for open-domain conversational agents. In Proceedings of the 42nd International ACM SIGIR Conference, pages 1273–1276.
  2. J. Ang, Y. Liu, and E. Shriberg. 2005. Automatic dialog act segmentation and classification in multiparty meetings. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)., volume 1, pages I/1061–I/1064 Vol. 1. find
  3. C. Bothe, C. Weber, S. Magg, and S. Wermter. 2018. A context-based approach for dialogue act recognition using simple recurrent neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 1952–1957, Miyazaki, Japan.
  4. E. Chapuis, P. Colombo, M. Manica, M. Labeau, and C. Clavel. 2020. Hierarchical pre-training for sequence labelling in spoken dialog. In Findings of the Association for Computational Linguistics: EMNLP, pages 2636–2648. find
  5. H. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. 2022. Scaling instruction-finetuned language models. CoRR, abs/2210.11416. ↗
  6. P. Colombo, E. Chapuis, M. Manica, E. Vignon, G. Varni, and C. Clavel. 2020. Guiding attention in sequence-to-sequence models for dialogue act prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7594–7601. ↗
  7. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. In Thirty-seventh Conference on Neural Information Processing Systems. J. Devlin, M-W.
  8. Chang, K. Lee, and K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT, 1:4171– 4186. J. St BT.
  9. Evans. 2003. In two minds: Dual-process accounts of reasoning. Trends in cognitive sciences, 7(10):454–459. find
  10. Z. Fan, Y. Gong, D. Liu, Z. Wei, S. Wang, J. Jiao, N.Duan, R. Zhang, and X. Huang. 2021. Mask attention networks: Rethinking and strengthen transformer. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1692–1701. ↗
  11. J. J. Godfrey and E. Holliman. 1993. Switchboard-1 Release 2 LDC97S62. Web Download. Philadelphia: Linguistic Data Consortium. find
  12. A. Goyal and Y. Bengio. 2022. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A, 478(2266):20210068. ↗
  13. G. Gronchi and F. Giovannelli. 2018. Dual process theory of thought and default mode network: A possible neural foundation of fast tinking. Frontiers in Psychology, 9(1237). find
  14. D. Hong, C. Feng, X. Zou, C. Hmelo-Silver, K. Glazewski, T. Wang, B. Mott, and J. Lester. 2024. Examining coordinated computer-based fixed and adaptive scaffolds in collaborative problem-solving game environments. In Proceedings of the Seventeenth International Conference on Computer-Supported Collaborative Learning, pages 43–50. Y. Hu, C-H. find
  15. Lee, T. Xie, T. Yu, N. A. Smith, and M. Ostendorf. 2022. In-context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Linguistics: EMNLP, pages 2627– 2643.
  16. D. Jurafsky, L. Shriberg, and D. Biasca. 1997. Switchboard SWBD-DAMSL shallow discourse-function annotation: Coders manual. Institute of Cognitive Science Technical Report, pages 97–102. find
  17. S. B. Kaufman. 2011. Intelligence and the cognitive unconscious. The Cambridge Handbook of Intelligence, pages 442–467. ↗
  18. S. Keizer, R. op den Akker, and A. Nijholt. 2002. Dialogue act recognition with Bayesian networks for dutch dialogues. In SIGdial Workshop on Discourse and Dialogue, pages 88–94.
  19. S. Kim, L. Cavedon, and T. Baldwin. 2010. Classifying dialogue acts in one-on-one live chats. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 862–871.
  20. Z. Kozareva and S. Ravi. 2019. ProSeqo: Projection sequence networks for on-device text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3894– 3903, Hong Kong, China. Association for Computational Linguistics. find
  21. H. Kumar, A. Agarwal, R. Dasgupta, and S. Joshi. 2018. Dialogue act sequence labeling using hierarchical encoder with CRF. In Proceedings of the AAAI conference on artificial intelligence, pages 3440–3447. ↗
  22. V. Kumaran, J. Rowe, B. Mott, S. Chaturvedi, and J. Lester. 2023. Improving classroom dialogue act recognition from limited labeled data with self-supervised contrastive learning classifiers. In Findings of the Association for Computational Linguistics, pages 10978–10992.
  23. G. Leech and M. Weisser. 2003. Generic speech act annotation for task-oriented dialogues. In Proceedings of the Corpus Linguistics 2003 Conference, volume 16, pages 441–446. Lancaster: Lancaster University. find
  24. Y. Levine, B. Lenz, O. Lieber, O. Abend, K. Leyton-Brown, M. Tennenholtz, and Y. Shoham. 2021. Pmi-masking: Principled masking of correlated spans. In Proceedings of the International Conference on Learning Representations (ICLR). ↗
  25. R. Li, C. Lin, M. Collinson, X. Li, and G. Chen. 2019. A dual-attention hierarchical recurrent neural network for dialogue act classification. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 383–392, Hong Kong, China. J. Lin, W. Tan, L. Du, Wr. ↗
  26. Buntine, D. Lang, D. Gašević, and G. Chen. 2024. Enhancing educational dialogue act classification with discourse context and sample informativeness. IEEE Transactions on Learning Technologies, 17:258–269. find
  27. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. RoBERTa: A robustly optimized BERT pre-training approach. CoRR, abs/1907.11692. ↗
  28. Z. Liu, Z. Wang, Y. Lin, and H. Li. 2022. A neural-symbolic approach to natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP, pages 2159–2172. find
  29. J. D. Marx and K. Cummings. 2007. Normalized change. American Journal of Physics, 75:87–91. find
  30. M. L. McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 23:276–282. ↗
  31. B. W. Mott, R. G. Taylor, S. Y. Lee, J. P. Rowe, A. Saleh, K. D. Glazewski, C. E. Hmelo-Silver, and J. C. Lester. 2019. Designing and developing interactive narratives for collaborative problem-based learning. In Interactive Storytelling: 12th International Conference on Interactive Digital Storytelling, ICIDS Proceedings 12, pages 86–100. Springer. find
  32. B. Noble and V. Maraev. 2021. Large-scale text pre-training helps with dialogue act recognition, but not without fine-tuning. In Proceedings of the International Conference on Computational Semantics (IWCS), pages 166–172.
  33. M. Nye, M. Tessler, J. Tenenbaum, and B. M. Lake. 2021. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. Advances in Neural Information Processing Systems, 34:25192–25204. ↗
  34. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. ↗
  35. V. Raheja and J. Tetreault. 2019. Dialogue Act Classification with Context-Aware Self-Attention. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3727–3733, Minneapolis, Minnesota. ↗
  36. S. Ravi and Z. Kozareva. 2018. Self-governing neural networks for on-device short text classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 887–893, Brussels, Belgium. Association for Computational Linguistics. find
  37. T. Saha, A. Patra, S. Saha, and P. Bhattacharyya. 2020. Towards emotion-aided multi-modal dialogue act classification. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 4361–4372, Online. Association for Computational Linguistics.
  38. A. Stolcke, K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky, P. Taylor, R. Martin, C. V. Ess-Dykema, and M Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗
  39. R. Sun. 2015. Interpreting psychological notions: A dual-process computational theory. Journal of Applied Research in Memory and Cognition, 4(3):191– 196. ↗
  40. D. Surendran and G.-A. Levow. 2006. Dialog act tagging with support vector machines and hidden markov models. In Proceedings of the 9th International Conference on Spoken Language Processing, pages 1950–1953. find
  41. A. Suresh, J. Jacobs, C. Harty, M. Perkoff, J. H. Martin, and T. Sumner. 2022. The talkmoves dataset: K-12 mathematics lesson transcripts annotated for teacher and student discursive moves. In Proceedings of the Conference on Language Resources and Evaluation, pages 4654–4662. find
  42. A. Suresh, J. Jacobs, V. Lai, C. Tan, W. Ward, J. H. Martin, and T. Sumner. 2021. Using transformers to provide teachers with personalized feedback on their classroom discourse: The talkmoves application. In the Spring AAAI 2021 Symposium on Artificial Intelligence for K-12 Education. ↗
  43. M. Syed and S. C. Nelson. 2015. Guidelines for establishing reliability when coding narrative data. Emerging Adulthood, 3:375–387. ↗
  44. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971. ↗
  45. A. A. Von Davier, M. Zhu, and P. Kyllonen. 2017. Innovative assessment of collaboration. Springer International Publishing. find
  46. Y. Xu, E. Yao, C. Liu, Q. Liu, and M. Xu. 2023. A novel ensemble model with two-stage learning for joint dialog act recognition and sentiment classification. Pattern Recognition Letters, 165:77–83. ↗
  47. M. Zhao, T. Lin, F. Mi, M. Jaggi, and H. Schütze. 2020. Masking as an efficient alternative to finetuning for pretrained language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2226–2241. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kim-2024-dual-process-masking-dialogue-act/